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This  thesis  presents  a  methodology  for  ineident  response  to  identify  anomalies  and 
malieious  adversary  persistence  within  the  networks  responsible  for  the  reliable  operation 
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the  historical  development  and  function  of  industrial  control  systems  (ICS)  and  their 
unique  security  issues.  The  study  of  public  technical  data  from  intrusions  into  control 
systems  produces  a  set  of  known  adversary  tactics  for  incorporation  into  the 
methodology.  This  work  further  documents  the  development  of  a  repeatable  technique  to 
collect  digital  forensic  artifacts  from  production  control  systems  that  is  compatible  with 
the  strict  operational  constraints  of  these  critical  networks.  The  technique  is  then  applied 
with  a  proof-of-concept  host-  and  network-based  toolkit  for  incident  response  that  is 
tested  against  real-world  data.  The  goal  of  the  methodology  and  the  supplementary 
toolkit  is  to  elicit  valuable,  previously-unavailable  findings  with  which  to  assess  the 
scope  of  malicious  intrusions  into  critical  ICS  networks. 
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I.  INTRODUCTION 


A,  BACKGROUND 

Throughout  history,  malicious  attackers  have  targeted  critieal  infrastructure  assets 
that  provide  nations  with  essential  services.  Around  600  BC,  Solon  of  Athens 
contaminated  the  water  supply  for  the  town  of  Cirrha,  allowing  Athens  to  swiftly  conquer 
the  town  while  the  Cirrhaens  were  violently  ill  [1],  Two  thousand  six  hundred  years  later, 
Vitek  Boden  of  Brisbane  maliciously  released  thousands  of  gallons  of  raw  sewage  into 
waterways  of  Maroochy  Island,  causing  significant  financial  and  economic  damages  and 
forcing  the  residents  to  relocate  [2],  [3].  In  both  cases,  a  single  malieious  actor  obtained 
unauthorized  access  to  critical  infrastructure  assets  and,  prior  to  detection,  caused  severe 
physical  consequences.  Now  critical  infrastructure  systems  are  more  susceptible  to 
malicious  attacks  than  ever  before. 

The  reliable  operation  of  modern  society’s  critical  infrastructure  depends  on 
industrial  control  systems  (ICS),  the  embedded  software  systems  that  allow  an  operator 
or  device  to  monitor  and  control  industrial  processes.  These  automation  systems  are 
ubiquitous  and  heterogeneous;  they  were  designed  with  much  implicit  trust  and  are  not 
very  compatible  with  most  modern  seeurity  solutions.  In  2010,  Stuxnet  demonstrated  that 
traditional  network  protections  like  segmentation  and  intrusion  detection  systems  (IDS) 
alone  are  insufficient  in  seeuring  control  systems  [4].  Insecurity  of  these  deviees  can  lead 
to  severe  consequences  and  malicious  hacking  tools  can  be  effeetive  without  much 
difficulty  to  an  attacker.  To  successfully  leverage  these  tools,  an  adversary  must  access 
the  trusted  network,  leaving  specific  forensic  artifacts  and  indicators  of  ICS  malicious 
activity. 

B,  MOTIVATION 

An  ICS  cyber  incident  response  process  is  not  well  developed  and  we  lack  tools 
built  specifically  for  identifying  current  or  historical  adversary  presence  within  the 
critical  systems  domain.  Few  published  efforts  reveal  actionable  teehnical  solutions  for 
ICS  seeurity  praetitioners  and  none  foeus  on  reliably  identifying  malieious,  persistent 

1 


access  within  live  data  from  production  ICS  devices.  The  primary  motivation  for  this 
thesis  was  based  on  technieal  gaps  observed  and  reported  [5]  during  the  course  of  ICS 
security  assessments  and  cyber  incident  response. 

Previous  studies  have  outlined  the  need  for  forensie  eolleetion  abilities  within  the 
ICS  environment.  The  Department  of  Homeland  Seeurity  (DHS)  Industrial  Control 
System  Cyber  Emergency  Response  Team  (ICS-CERT)  has  eoneluded  that  traditional 
forensie  tools  are  not  suitable  for  use  in  ICS  networks  [6].  ICS-CERT  provides  a  high- 
level  strategy  for  forensics,  makes  the  ease  for  having  an  ineident  response  capability, 
and  presents  a  breakdown  of  what  is  different  about  ICS  systems.  However,  the 
ICS-CERT  best  praetiee  doeuments  only  reeommend  that  the  capability  should  be  ereated 
but  the  papers  do  not  offer  speeifie  tools  and  teehniques  to  implement  ineident  response. 
Some  researeh  has  posited  that  live  forensies  on  automation  networks  is  feasible; 
however  no  speeifie  adversary  identification  techniques  have  been  provided  and 
no  available  forensie  toolkits  are  available  that  have  been  designed  for  use  on  ICS 
networks  [7]. 

C.  PROBLEM  STATEMENT 

Critieal  infrastrueture  owners  and  operators  are  poorly  equipped  to  discover  and 
respond  to  intrusions  into  ICS  networks.  Effeetive,  eompatible  tools  do  not  exist  to 
reliably  extraet  the  neeessary  teehnieal  data  to  analyze  ICS  environments  with  modem 
ineident  response  techniques  [7].  Adversaries  eonstmct  surreptitious  pathways  to  their 
target  systems  that  provide  persistent  aeeess  for  reeonnaissanee  and  future  malieious 
aetivity.  These  adversary  persistenee  meehanisms  often  remain  undeteeted  for  signifieant 
periods  of  time.  Seeurity  teams  require  a  repeatable,  tailored  response  methodology 
employing  host  and  network  data  eolleetion  and  analysis  teehniques  to  identify  malieious 
pathways  and  adversary  presence  on  ICS  networks.  The  methodology  and  supplementary 
toolkit  within  this  thesis  represent  a  step  toward  addressing  that  need. 
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D,  APPROACH  AND  LIMITATIONS 


This  thesis  proposes  a  structured  methodology  to  identify  malicious  activity  by 
using  host-based  forensics  and  network  analysis  to  identify  anomalous  client-side  attack 
vectors  during  ICS  assessments  and  incident  response. 

1,  Scope 

This  thesis  will  cover  analysis  of  control  systems  that  respond  to  command-line 
forensic  interrogation  techniques  and  that  communicate  over  accepted,  interpretable  ICS 
protocols.  It  will  not  cover  embedded  device  firmware  extraction  or  offline  drive  image 
analysis  within  the  host-based  methodology.  ICS  field  devices  will  be  discussed,  but 
direct  input/output  (I/O)  protocols  and  serial  communication  are  not.  This  thesis  also  does 
not  cover  response  actions  should  adversary  presence  be  detected,  nor  disaster  recovery, 
although  specific  case-by-case  suggestions  will  be  made. 

2,  Environment  and  Data  Availability 

To  develop  the  most  robust  and  reliable  methodology,  real  data  from  ICS 
networks  will  be  used  in  this  study.  Some  experimentation  may  be  conducted  through  the 
course  of  regular  assessments  and  incident  response  with  critical  infrastructure  asset- 
owner  approval.  Multiple  regulations  protect  disclosure  of  this  data;  anonymized  data 
will  be  used  when  possible,  to  include  host-based  artifacts  and  scrubbed  ICS  network 
traffic.  Additional  experiments  will  be  conducted  within  a  closed  network  that  replicates 
real  ICS  architectures. 

E.  THESIS  ORGANIZATION  AND  CONTRIBUTIONS 

Chapter  II  provides  background  on  ICS  systems  and  security.  Chapter  III  contains 
technical  case  studies  of  relevant  cyber  incidents.  Chapter  IV  describes  a  methodology 
for  identifying  malicious  activity.  Chapter  V  presents  the  experiments  and  findings  of  this 
study.  Chapter  VI  provides  a  summary  and  recommends  areas  for  future  work. 
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II.  BACKGROUND 


A,  INDUSTRIAL  CONTROL  SYSTEMS 

The  reliable  operation  of  modern  soeiety’s  critieal  infrastructure  depends  on  the 
proper  functioning  of  industrial  control  systems.  These  vital  systems  are  widely 
implemented  across  all  infrastructure  sectors  to  automate  large  industrial  processes. 

Components  of  ICS  networks  are  designed  to  allow  embedded  logic  to  control  a 
process  efficiently  without  constant  human  intervention,  and  thus  these  components  have 
specific  roles  and  constraints  to  operate  as  low-level  building  blocks  for  industrial 
automation. 

1.  Industrial  Control  Systems  History 

In  1959,  a  Texaco  refinery  in  Port  Arthur,  Texas  installed  the  first  ICS  device  in 
history,  a  Thompson  Ramo  Wooldridge  RW-300  direct  control  process  computer  [8]. 
Control  systems  were  originally  designed  to  monitor  and  control  industrial  processes  in 
complete  isolation,  much  like  server  mainframes  were  initially  configured.  One  central 
master  unit  provided  all  computing,  control,  and  monitoring  functionality,  quietly 
executing  simple  instructions  or  ladder  logic  [9].  Fault  tolerance  was  achieved  through 
complete  system  redundancy:  A  duplicate  ICS  master  unit  provided  all  functions  of  the 
original  and  monitored  the  primary  ICS  system’s  operation.  If  the  secondary  system 
detected  a  fault,  it  commandeered  all  operations.  Vendors  continued  to  follow  this 
mainframe  architecture  configuration  for  ICS  systems  through  the  1960s  and  continuing 
into  the  early  1980s,  installing  unique  systems  with  proprietary  protocols  as  automation 
technology  matured. 

As  personal  computer  (PC)  costs  became  less  prohibitive  and  Local  Area 

Network  (LAN)  technology  was  embraced  for  business  networks  in  the  late  1980s, 

individual  computers  began  to  replace  static  ICS  components  which  enabled  distributed 

functionality  and  processing  across  multiple  control  systems  [9].  Data  historians  and 

human-machine  interfaces  (HMIs)  were  implemented  not  as  standalone  systems  but  as 

vendor-specific  proprietary  software  on  specialized  personal  computers.  The  data 
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historian  is  a  database  that  contains  a  history  of  the  state  of  a  process-control  system  [10]. 
HMIs  still  provide  the  graphical  user  interface  (GUI)  front-end  for  operators  today  and 
continue  to  be  one  of  the  few  human-to-machine  touchpoints  in  the  largely  automated 
control  environment.  These  computer-based  processes  ran  modified  versions  of  common 
operating  systems  and  were  generally  vendor-provided  systems,  with  only  the  vendor 
able  to  provide  upgrades  and  system  maintenance.  As  a  result  of  the  newly  distributed 
architecture,  the  ICS  domain  no  longer  required  full-time  redundant  failover  systems 
because  the  PC-based  ICS  systems  could  share  the  operations  load  of  failed  components. 

In  the  late  1990s,  vendors  started  to  embrace  commercial  off-the-shelf  computer 
systems  and  networking  hardware.  These  vendors  began  to  adapt  their  legacy  proprietary 
protocols  for  field  devices  so  that  the  systems  could  communicate  over  the  Internet 
protocol  suite  (TCP/IP).  The  last  section  of  this  chapter  examines  ICS  protocols  more 
comprehensively.  Around  this  time,  Microsoft  Windows  became  the  operating  system  of 
choice  for  HMIs,  engineering  workstations,  and  several  ICS  servers.  Vendors  used  old  or 
modified  versions  of  Windows  that,  once  initially  tested  for  compatibility  with  vendor 
equipment,  were  rarely  updated. 

These  newly  networked  and  communicating  field  devices,  like  remote  terminal 
units  (RTUs)  and  programmable  logic  controllers  (PLCs),  allowed  for  customized 
implementations.  RTUs  are  field  devices  that  transmit  telemetry  data  (information  that  is 
collected  at  remote  or  inaccessible  points)  to  master  systems.  RTUs  also  accept 
commands  from  those  master  systems  to  control  connected  objects.  PLCs  are  also  field 
devices  and  they  execute  a  wide  array  of  programmable  functions  such  as  vibration 
monitoring,  catalyst  loading,  area  monitoring,  and  product  loading/unloading.  PLCs 
generally  execute  ladder  logic  but  now  can  run  higher  level  programming  languages  like 
C++;  they  offer  the  lowest-level  code  before  reaching  physical  systems.  With  the 
introduction  of  these  field  devices,  the  ICS  component  manufacturers  enabled  the  system- 
integrators  and  critical  infrastructure  companies  to  make  their  own  modifications  and 
leverage  their  existing  network  infrastructure.  Companies  have  continued  to  develop 
networked  ICS  architectures  that  allow  for  streamlined  performance  tracking,  accurate 
billing,  and  off-site  backup  capabilities,  despite  the  growing  security  concerns  that  this 
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interconnection  has  introduced.  This  fully  distributed  and  networked  architecture  is 
necessary  for  industrial  applications  that  require  distributed  monitoring  points,  such  as 
electric  power  transmission  and  distribution,  oil  and  gas  pipeline  and  production 
operations,  and  water  utility  operations. 

The  centralized  consolidation,  processing,  and  management  of  the  field  devices 
spawned  another  system  term  “SCADA”  which  is  often  incorrectly  used  to  represent 
the  larger  set  of  ICS  components.  Supervisory  control  and  data  acquisition  (SCADA) 
systems  extend  the  capabilities  of  an  automated  system  so  that  it  can  be  monitored 
and  even  controlled  from  a  remote  location  [11].  The  implementation  of  SCADA  systems 
allows  for  the  control  of  industrial  processes  across  widespread  geographical 
locations  and  facilitates  the  fetching  and  presentation  of  current  data  values  to  a  human 
operator  [12]. 

2,  Industrial  Control  Network  Composition 

All  automation  components  that  have  been  developed  over  the  past  fifty  years 
(SCADA,  HMIs,  RTUs,  and  PLCs)  are  implemented  in  some  form  in  most  ICS  network 
configurations.  SCADA  systems  continue  to  manage  very  large-scale  processes  at 
multiple  sites  and  over  large  distances.  HMIs  are  the  devices  which  present  process  data 
to  their  human  operators,  who  control  and  monitor  the  processes.  Some  common  HMIs  in 
industry  include  Wonderware,  Siemens  WinCC,  Rockwell  RSView,  and  Areva’s  e-terra 
[10].  RTUs  interconnect  all  the  sensors  in  the  process  and  also  convert  those  sensor 
signals  to  digital  data,  which  is  routed  to  the  SCADA  system.  PLCs  are  field  devices  that 
are  cheap,  flexible,  and  highly-configurable.  When  combined  within  an  industrial  control 
network,  these  devices  retain  features  of  the  individual  system,  leading  to  unique, 
expensive  configurations  that  have  the  functionality  and  weaknesses  of  the  legacy 
equipment. 

ICS  components  can  be  interconnected  with  a  variety  of  mediums.  Direct-wiring 
is  common  for  intra-facility  connections.  Microwave  communications  backbones  are  very 
frequently  used  between  facilities,  especially  in  oil  and  natural  gas  pipelines,  but  fiber 
optic  inter-facility  deployments  are  becoming  more  common.  Both  spread-spectrum  and 


7 


narrow-band  radio  are  used  to  connect  remote  ICS  components,  however  dial-up  or 
cellular  modems  continue  to  provide  connection  to  these  systems. 

As  explained  above,  there  are  many  components  and  countless  possible 
configurations.  Attempting  to  broadly  capture  several  possible  ICS  network  architectures 
produces  a  complex  diagram,  such  as  Figure  1 . 


Figure  1.  General  ICS  Network  Composition  Diagram,  from  [6] 

While  this  is  helpful  for  understanding  the  complexities  of  these  systems,  a  more 
abstracted  model  can  be  used  to  encapsulate  the  myriad  of  components  and 
configurations.  Researchers  introduced  an  abstracted  ICS  architecture,  shown  in  Figure  2, 
which  identifies  the  key  functions  within  the  ICS  domain  [10]. 
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Figure  2.  Abstracted  ICS  Network  Diagram,  from  [10] 


At  the  top  right  of  Figure  2,  the  traditional  IT  network,  which  hosts  the  corporate 
network  and  controls  site  manufacturing  operations,  is  interconnected  to  the  rest  of  the 
ICS  network  by  one  of  several  domain- interconnection  methods  discussed  in  the  next 
subsection.  Inside  the  firewall  in  Figure  2,  the  control  center  network  houses  the 
supervisory  services  that  reside  on  the  engineering  workstation,  SCADA  input/output 
(I/O)  server,  and  the  HMI.  These  supervisory  control  systems  connect  using  ICS 
protocols  over  Ethernet  to  the  field  devices  such  as  PECs  or  RTUs  that  are  often  located 
at  distributed  sites.  The  field  devices  receive  commands  from  the  supervisory  systems 
that  instruct  the  field  devices  to  subsequently  control  or  acquire  data,  often  over  direct 
I/O,  from  the  physical  ICS  assets  like  mechanical  valves,  circuit  breakers,  voltage 
regulators,  digital  temperature  sensors,  or  other  smart  devices.  The  data  acquired  from  the 
physical  ICS  assets  is  transferred  from  the  PECs  and  RTUs  to  a  central  control  center 
where  it  is  displayed  to  the  operator  using  an  HMI. 
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3. 


ICS  Domain  Interconnection  Methods 


In  addition  to  the  distribution  and  communication  paths  between  ICS  components, 
there  are  often  additional  network  interconnections  between  the  traditionally  isolated  ICS 
domain  and  the  corporate  network.  These  interconnections  are  introduced  for  a  variety  of 
reasons.  The  ICS  network  contains  valuable  information  for  business  applications,  such 
as  billing  and  financial  data,  equipment  trending,  and  operational  reports. 
Interconnections  to  external  networks  may  have  been  introduced  to  allow  for  remote 
vendor  support  or,  in  the  case  of  the  inter-control  center  communications  protocol 
(ICCP),  for  utilities  to  share  electrical  power  status  for  grid  stability  [14].  Remote  access 
technology  allows  for  access  to  corporate  software  from  field  locations  and  provides 
capability  to  manage  devices  that  are  difficult  to  access. 

Dedicated  lines  are  expensive,  while  the  Internet  is  cheap  and  pervasive. 
According  to  the  National  Institute  of  Standards  and  Technology  (NIST),  “widely 
available,  low-cost  Internet  protocol  (IP)  devices  are  now  replacing  proprietary  solutions, 
which  increases  the  possibility  of  cybersecurity  vulnerabilities  and  incidents”  [15].  Low 
cost  and  easy-to-install-and-maintain  wireless  connectivity  has  been  added  to  field 
devices,  allowing  for  the  bypass  of  the  physical  security  boundary  and  direct  connection 
of  field  devices  to  the  Internet. 

ICS  inter-domain  networking  can  be  architected  using  a  variety  of  methods,  such 
as  explicit  direct  connections,  firewall-controlled  connections,  demilitarized  zone  (DMZ) 
only  connections,  or  data  diodes.  Direct  connections  can  be  hard  to  trace  as  they  are 
constructed  by  employees  or  vendors  uniquely  for  the  site,  usually  with  standard 
protocols  such  as  secure  shell  (SSH),  Telnet,  fide  transfer  protocol  (FTP),  virtual  private 
network  (VPN),  or  with  dial-up  connections.  This  means  that  the  direct  connections  may 
not  be  known  to  the  security  team.  Firewalls  and  access  control  lists  (ACLs)  are  often 
used  for  this  interconnection  and  only  allow  certain  types  of  connections;  but  the 
software  for  them  provides  only  limited  support  for  ICS  protocols.  DMZs  are  common 
with  these  interconnections;  however  they  should  not  be  constructed  to  have  access  to  the 
Internet  like  traditional  web  server  DMZs.  When  properly  implemented,  isolated 

networks  can  communicate  with  the  DMZ  but  not  with  one  another,  making  the  DMZ 
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good  for  storage  servers  such  as  databases  and  data  historians.  Data  diodes  enforce 
unidirectional  network  flow,  making  them  ideal  for  the  ICS  environment,  but  they  do  not 
allow  for  acknowledgement  or  response  and  thus  are  not  fully  compatible  with  the 
transmission  control  protocol  (TCP).  This  means  that  data  diodes,  although  difficult  to 
implement  with  many  protocols,  can  address  a  limited  set  of  inter-domain  communication 
needs.  Regardless  of  interconnection  type,  a  single  control  such  as  a  DMZ,  firewall,  or 
data  diode  cannot  provide  sufficient  defense  alone.  Since  there  are  no  guaranteed 
defenses  for  business  networks  and  almost  all  are  interconnected  with  ICS  networks  in 
some  way,  business  network  incidents  can  intentionally  or  collaterally  affect  ICS 
networks. 

It  is  believed  that,  in  2013,  part  of  the  Austrian  and  German  electric  power  grid 
almost  failed  due  to  a  single  misconfigured  ICS  network  interconnection  when  a  status 
request  command  packet,  sent  from  a  German  gas  company  as  a  test  of  their  new 
equipment,  made  its  way  onto  an  Austrian  energy  power  control  and  monitoring  network 
[16].  Once  on  the  Austrian  ICS  network,  the  message  generated  thousands  of  reply 
messages  and  flooded  the  control  network.  In  order  to  resolve  the  self-inflicted 
distributed  denial-of-service  (DDoS)  incident  prior  to  power  outages,  a  portion  of  the 
monitoring  and  control  network  had  to  be  isolated  and  disconnected  [16].  Thus,  it  is  clear 
that  not  only  the  interconnection  medium  must  be  understood  but  also  the  communication 
protocols  used  by  the  ICS  devices. 

4,  ICS  Communication  Protocol  Inspection 

Since  the  development  of  ICS  technology  was  vendor-driven  with  a  wide  variety 
of  competing  hardware,  software,  and  capabilities,  the  communication  technology  was 
similarly  developed  in  a  loose,  ad-hoc  fashion  and  lacked  a  central  standards  body.  The 
resulting  list  of  communication  protocols  include  Modbus  and  the  distributed  network 
protocol  revision  3  (DNP3)  as  well  as  hundreds  of  proprietary  protocols  like  ANSI 
X3.28,  CDC  Types  1  and  2,  Conitel  2020/2000/3000,  DCP  1,  Gedac  7020,  IBM  3707, 
ICCP,  lEC  61850,  Landis  &  Gyr  8979,  OPC,  Redac  70H,  Tejas  3  and  5,  TRW  9550,  and 
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UCA.  In  total,  there  are  estimated  to  be  between  150  and  200  different  ICS  protocols 
[IV]. 

Despite  the  fragmented  proprietary  protocols  and  slow  adoption  rate  of  new 
technology  in  ICS,  Modbus,  DNP3,  and  the  Ethernet  industrial  protocol  (Ethernet/IP)  are 
the  most-used  ICS  protocols  [18].  The  Modbus  messaging  protocol  was  developed  by 
Modicon  in  1979  to  establish  client-server/master-slave  communication  between 
intelligent  devices,  first  over  serial  lines  then  later  incorporating  TCP  [19]  (the  TCP  suite 
are  the  most  common  protocols  on  the  Internet).  DNP3  was  developed  as  a  set  of 
protocols  for  data  acquisition  and  control  equipment  communication,  and  in  2010  the 
Institute  of  Electrical  and  Electronics  Engineers  (IEEE)  established  DNP3  as  the  standard 
for  electrical  power  system  communications  (IEEE  1815)  [20].  Easily,  Ethemet/IP  was 
originally  developed  by  Rockwell  Automation  in  2001  but  is  now  managed  by  the  Open 
DeviceNet  Vendors  Association  (ODVA)  and  is  an  application-layer  protocol  that 
implements  the  common  industrial  protocol  (CIP)  over  TCP  [21]. 

ICS  protocols  have  multiple  telemetry  schemes  such  as  reporting  by  exception 
(which  is  common  in  Europe),  in  round-robin  communications,  or  at  a  time  polling 
interval.  Most  of  these  protocols  are  primitive  and  field  devices  cannot  be  reliably 
queried  to  see  what  protocols  they  support.  All  these  factors  result  in  a  highly-complex 
forensic  and  incident  response  process.  There  is  a  recent  trend  toward  routable,  industry- 
standard  protocols  that  may  someday  replace  these  legacy  vendor-specific  proprietary 
protocols  [15];  however  the  long  product  life  cycle  and  high  cost  for  replacing  ICS 
components  means  this  standardized  environment  is  a  long  way  off. 

Despite  their  differences,  every  ICS  protocol  functions  using  master-slave 

communication.  The  master  polls  for  data,  controls  slave  devices,  and  maintains  a 

repository  of  data.  The  slaves,  transmitting  either  by  polling  or  reporting  by  exception, 

respond  to  master  commands.  Although  all  components  in  the  ICS  domain  are  either  a 

master  or  a  slave,  it  is  important  to  note  that  slaves  can  have  more  than  one  master,  and  a 

device  can  be  a  master  in  one  environment  and  a  slave  in  another  as  in  a  tiered 

architecture.  This  tiered  structure,  with  a  master  at  a  remote  site  gathering  all  data  for 

transmission  to  the  next  hop,  saves  bandwidth  and  reduces  the  poll  cycle.  Also  important 
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to  this  study  is  that  most  modem  protocols  researched  communicate,  albeit  via  simple 
wrappers  and  often  in  cleartext,  using  TCP/IP  for  subprotocols,  which  allows  for  some 
level  of  passive  network  traffic  parsing. 

B.  INDUSTRIAL  CONTROL  SYSTEMS  SECURITY 

As  explained  in  the  previous  section,  ICS  design  allows  operators  to  interact  with 
the  control  systems  at  an  abstract  level,  significantly  lowering  the  technical  expertise  and 
knowledge  of  the  system  required  to  keep  the  process  mnning.  The  resulting  state  of  ICS 
network  security  is  that  field  devices  are  low  capability,  designed  for  performance  rather 
than  security,  and  operate  on  a  large  number  of  unique  protocols  that  must  all  be 
protected  equally  [17].  ICS  systems  were  built  to  be  highly  available  machines  and 
integrity  and  confidentiality  were  afterthoughts  That  is,  ICS  hardware  was  never 
designed  with  security  in  mind  because  they  were  originally  deployed  in  isolated 
environments,  removed  from  external  networks.  As  those  systems  have  become 
increasingly  connected  to  the  network  across  much  of  critical  infrastmcture,  ICS  systems 
are  more  accessible  and  susceptible  to  malicious  attacks.  “Control  systems  are  the 
‘brains’  of  the  control  and  monitoring  of  the  bulk  electric  system  and  other  critical 
infrastmctures,  but  they  were  designed  for  functionality  and  performance,  not  security. 
Most  control  systems  assume  an  environment  of  complete  and  implicit  trust”  [23]. 

I,  Unique  ICS  Security  Vulnerabilities 

The  same  control  systems  that  replaced  so  many  human  functions,  such  as 
flipping  a  switch  or  turning  a  knob  based  on  automated  thresholds,  also  introduced  a 
range  of  security  concerns.  The  major  vulnerabilities  for  ICS  are  in  three  areas:  the 
prevalence  of  legacy  equipment,  the  requirement  for  real-time  availability,  and  the 
patching  difficulties  associated  with  partially  or  fully  segmented  networks. 

a.  Legacy  Equipment  Challenges 

The  product  life  cycle  for  ICS  equipment  is  considerably  longer  than  traditional 
information  technology  (IT)  systems.  20-year-old  systems  are  common  compared  to  3-to- 
5-year-old  systems  found  in  traditional  IT  networks.  Due  to  the  cost  of  individual 
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components,  it  is  not  feasible  to  replaee  all  inseeure  legaey  equipment  and  these 
ehallenges  are  eommon  aeross  all  of  eritieal  infrastrueture’s  ICS  deployments.  Vendor 
support  is  limited  eompared  to  traditional  eomputers  with  few  support  styles  and  often  a 
single  vendor  supporting  many  systems.  Forensie  eapabilities  are  laeking  on  most  ICS 
equipment  sinee  field  deviees  do  not  generally  store  logs  and  their  alarms  and  responses 
ean  be  suppressed. 

To  eommunieate  in  these  mixed  environments,  with  both  legaey  and  modem 
equipment,  proprietary  ICS  protoeols  were  modified  to  be  used  in  IP -based  networks. 
The  result  is  already  weak  protoeols  transformed  into  eleartext  paekets  wrapped  in 
TCP/IP  layers,  whieh  has  only  inereased  the  pre-existing  attaek  surfaee.  With  prevalent 
eleartext  eommunieations  and  limited  authentieation  or  validation,  ICS  networks  provide 
ample  opportunity  for  tampering,  intereeption,  and  injeetion  of  data.  With  persistent 
aeeess  to  the  ICS  supervisory  loeal  area  network,  an  adversary  eould  eonduet  an 
eavesdropping  attaek  on  the  SC  ADA  server’s  eommunieation  with  PLCs,  sinee  this  is 
often  in  eleartext  and  eould  be  manipulated.  This  eould  enable  injeeted  telemetry  data, 
adversary  knowledge  of  system  events,  enumeration  of  equipment,  or  loss  of  market¬ 
relevant  information.  From  a  seeurity  perspeetive,  most  ICS  traffie  is  aeeepted  if 
addresses  mateh  and  eyelie  redundaney  eheeks  (CRCs)  validate. 

Modern  solutions  like  eneryption  are  often  not  implemented  and  frequently  are 
not  supported  within  legaey  ICS  protoeols.  In  faet,  the  popular  Modbus  and  DNP3 
protoeols  eurrently  do  not  support  authentieation,  integrity  eheeking,  authorization,  or 
eneryption.  Third-party  seeurity  solutions  may  not  be  applieable  sinee  the  eomponents 
were  designed  to  support  the  intended  industrial  proeess  and  the  eomponents  may  not 
have  enough  eomputing  resourees  or  memory  to  support  the  addition  of  seeurity 
eapabilities.  The  later  that  seeurity  is  eonsidered  in  the  development  of  many  deviees,  the 
more  diffieult  and  expensive  its  seeurity  ean  beeome.  This  problem  is  expeeted  to 
eontinue  for  some  time,  sinee  the  long  lifespan  of  ICS  eomponents  requires  that  even  new 
equipment  added  to  the  network  will  have  to  be  baekwards-eompatible  with  teehnology 
or  protoeols  that  are  10-or-more  years  old  [24]. 
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b.  Real-time  Availability  Requirements 

Any  solution  attempting  to  address  the  legacy  equipment  vulnerabilities  must  not 
introduce  latency  or  overhead  into  the  network  traffic,  especially  0%  downtime  critical 
traffic  that  must  be  operational  24  hours  a  day  for  365  days  a  year. 

An  ICS  network  has  minimal  tolerance  for  communication  delay  or  data  loss.  The 
environment  is  expected  to  be  available  for  extended  periods  of  time  and  to  meet  strict 
timing  requirements  [14].  The  “flipped  CIA  triad”  is  often  referenced  by  ICS  security 
experts  [25].  Whereas  traditional  IT  systems  are  built  around  confidentiality,  integrity, 
and  availability  (CIA  triad),  in  that  order,  control  systems  are  the  opposite.  They  were 
designed  with  the  requirement  that  real-time  availability  comes  first,  then  the  integrity  of 
the  telemetry  data,  and  lastly  confidentiality. 

Real-time  availability  requirements  were  historically  addressed  with  redundancy 
[25]  but  full-time  redundant  backups  are  no  longer  the  industry  standard.  Additionally, 
increased  competition  within  multiple  critical  infrastructure  industries  and  a  focus  on  cost 
control  has  led  to  increased  asset  use  with  fewer  trained  staff  responsible  to  maintain  the 
systems’  uptime. 

With  the  same  uptime  requirements  but  without  fully  redundant  backups,  simple 
IT  functions  like  rebooting  may  not  be  acceptable  due  to  process  availability 
requirements.  The  throughput  and  uptime  requirements  of  these  networks  create  an 
environment  where  continuous  reliable  operations  will  always  trump  security 
assessments,  forensics,  and  incident  response. 

c.  Problematic  Patching 

ICS  components  tend  to  be  inadequately  patched  compared  to  IT  systems.  The 

security  afforded  by  any  amount  of  ICS  network  segmentation  is  exchanged  for  the 

ability  to  easily  manage  patches,  antivirus  definitions,  and  firmware  updates.  Patch 

management  is  essential  to  update  or  repair  components  of  systems  that  have  identified 

vulnerabilities  affecting  the  validity  and  integrity  of  device  operation.  However,  patching 

of  ICS  software  in  critical  infrastructure  is  “inconsistent  at  best  and  non-existent  at 

worst”  according  to  NIST  [15].  This  is  compounded  by  the  lack  of  vendor  support  and 
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the  difficulty  of  upgrading  within  the  highly  volatile  and  sometimes  unstable 
environment  in  which  control  systems  reside.  Months  of  planning  is  often  required  in 
order  to  take  offline  and  apply  patches  to  production  ICS  systems.  Critical  infrastructure 
asset  operators  must  weigh  the  planning  difficulties  and  costs  of  patching  ICS  systems  for 
stable  and  accurate  operations  against  the  possibility  of  malicious  tampering  with  that 
component.  87  percent  of  ICS-specific  vulnerabilities  reported  to  DHS  in  2013  were 
exploitable  remotely  over  the  network  [26].  Furthermore,  even  if  patches  are  scheduled 
and  applied  properly,  they  can  introduce  instability  into  the  ICS  domain  if  not  thoroughly 
tested. 

This  means  that  a  growing  number  of  “forever  day”  vulnerabilities  are  being 
discovered  in  older  control  systems  [27].  These  vulnerabilities,  unlike  zero-day  exploits 
where  vendors  and  security  firms  can  deploy  patches,  exist  on  the  legacy  equipment 
described  above  and  can  be  targeted  specifically  with  the  knowledge  that  they  are 
weaknesses  on  older  control  systems  that  are  not  continually  supported. 

2.  Consequences  of  ICS  Failure 

NIST  states  that,  “ICS  are  typically  used  in  industries  such  as  electrical,  water  and 
wastewater,  oil  and  natural  gas,  chemical,  transportation,  pharmaceutical,  pulp  and  paper, 
food  and  beverage,  and  discrete  manufacturing  (e.g.,  automotive,  aerospace,  and  durable 
goods)”  [15].  These  systems  directly  control  the  processes  that  operate  and  stabilize  our 
critical  infrastructure,  including  everything  from  dams  and  water  treatment  plants  to 
electric  power  utilities  and  nuclear  generation  plants.  The  failure  of  control  systems  can 
directly  manifest  in  the  physical  world  with  effects  ranging  from  regulatory  non- 
compliance  to  severe  physical  disasters. 

Since  ICS  systems  are  crucial  to  the  operation  of  the  electric  grid,  their  failure 
could  lead  to  blackouts,  economic  disruption,  and  loss  of  life  [15].  In  the  electricity 
sector,  instability  can  lead  to  cascading  outages  and  loss  of  communication,  especially 
power  systems  with  dynamic  and  dramatic  changes  in  load.  In  that  scenario,  there  is  a 
complex,  multi-step  restoration  process.  First,  critical  loads  like  hospitals  must  have 
power  restored,  then  generation  facilities,  transmission  lines,  distribution  feeders,  and 
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lastly  corporate  and  residential  power  serviee.  In  the  oil  and  natural  gas  industry,  system 
failure  ean  result  in  fuel  shortages  with  eeonomie  impaets  and  even  explosions,  eausing 
loss  of  human  life  or  environmental  catastrophes.  In  the  water  sector,  improper  treatment 
or  eontamination  of  water,  release  of  waste,  or  redueed  pressure  to  emergeney  systems 
like  fire  hydrants  are  possible  eonsequenees.  Chemieal  ICS  systems  ean  be  tampered  with 
resulting  in  ehemieal  formula  manipulation.  Even  the  transportation  seetor  relies  heavily 
on  ICS  networks  for  eontrol  of  mass  transit,  and  their  failure  eould  lead  to  train 
derailments  or  erashes. 

The  hardware  and  software  for  ICS  eomponents  is  almost  exelusively  foreign- 
owned  [29]  and  there  are  a  limited  number  of  eritieal  asset  manufaeturers,  with  long  lead 
times  required  for  high-value  eomponents.  For  example,  it  ean  take  six  months  to  reeeive 
a  replaeement  for  eertain  transformers.  The  United  States’  eritieal  infrastrueture  is 
often  referred  to  as  a  “system  of  systems”  because  of  the  interdependeneies  that  exist 
between  its  various  industrial  sectors  as  well  as  intereonneetions  between  business 
partners  [30],  [31]. 

The  failures  of  these  systems  ean  be  very  real.  In  June  1999,  the  Olympie  Pipeline 
Company’s  pipeline  ruptured  eausing  massive  physieal  eonsequenees,  ineluding  three 
deaths  and  multiple  additional  injuries  ineluding  hydroearbon  inhalation  poisoning  [32]. 
The  National  Transportation  Safety  Board  (NTSB)  coneluded  that  the  SCADA  serviees 
failed  on  its  single  Ethernet  baekbone  and  event  logs  indicated  that  the  SCADA  system 
failed  to  exeeute  the  eommand,  partially  as  a  result  of  database  development  work  being 
done  on  the  SCADA  system  while  it  was  being  used  to  operate  the  pipeline,  whieh  led  to 
the  systems  non-responsiveness  during  eritieal  operations  [32].  Had  the  supervisory  ICS 
eomponents  remained  responsive  to  the  eommands  of  the  Olympie  eontrollers,  the 
eontroller  operating  the  aeeident  would  have  been  able  to  initiate  aetions  that  would  have 
prevented  the  pressure  increase  that  ruptured  the  pipeline. 

In  August  2003,  the  northeastern  region  of  the  United  States  and  Canada 

experieneed  a  blaekout  that  affeeted  50  million  people  due  to  caseading  failures  of 

eleetrieal  grid  operation.  While  the  root  eause  of  the  blaekout  was  tree  limbs  eontacting 

transmission  wires,  it  was  later  determined  that  a  previously  unknown  software  bug  in 
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GE  Energy’s  XA/21  system  at  EirstEnergy  Corporation  in  Akron,  Ohio  was  partially 
responsible  for  the  initial  breakdown  when  it  silently  failed  and  did  not  trigger  the  alarm 
system  [33]. 

In  Mareh  2008,  an  engineer  at  the  Hateh  nuelear  power  plant  installed  a  software 
update  on  a  eomputer  operating  within  the  plant’s  business  network.  When  that  eorporate 
network  eomputer  rebooted,  it  reset  the  data  on  the  eontrol  system,  eausing  safety  system 
to  misinterpret  the  laek  of  data  as  a  drop  in  eoolant  water  reservoirs  and  a  plant  shutdown 
automatieally  began  [34].  The  nuelear  plant  shutdown  took  48  hours  to  reeover. 
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III.  CASE  STUDY  TECHNICAL  ANALYSIS 


A,  CATEGORIES  OF  MALICIOUS  ACTIVITY 

To  identify  malicious  activity  in  the  ICS  domain,  it  is  necessary  to  review  case 
studies  of  previous  intrusions.  Due  to  the  limited  availability  of  public  data  on  past  ICS 
incidents,  this  thesis  categorizes  similar  events  together  to  isolate  behavioral  trends.  Past 
malicious  activity  related  to  control  systems  can  be  assigned  to  one  of  three  categories: 
custom  targeted  exploits,  commodity  malware  in  the  control  environment,  and 
unauthorized  persistent  ICS  network  access. 

1.  Custom  Targeted  Exploits 

This  category  of  malicious  activity  includes  nation-state-sponsored  hacking  or 
other  highly-targeted  and  well-funded  intrusions,  often  using  custom  exploits.  These 
attacks  have  targeted  PLCs  and  other  ICS  field  devices  and  can  involve  firmware 
tampering  or  exploiting  other  device  vulnerabilities.  Targeted  attacks  such  as  those 
sponsored  by  nation  states  tend  to  use  zero-day  exploits  that  have  not  been  publicly 
disclosed  or  patched  by  the  vendor.  Historical  examples  of  targeted  critical  infrastructure 
attacks  include  the  proof-of-concept  Aurora  exploit,  Shamoon,  and  most  notably  Stuxnet. 

In  early  2003,  a  marine  terminal  in  Venezuela  was  the  target  of  attempted 
sabotage.  The  technical  details  have  not  been  shared  publicly,  but  it  is  believed  that  a 
team  of  hackers  obtained  access  to  the  SCADA  network  of  the  oil  tanker  loading 
machinery  and  overwrote  PLCs  with  an  empty  program  module  [16].  The  result  was  an 
eight  hour  halt  of  the  oil  tanker  loading  process  until  the  backup  ladder  logic  was 
reinstalled  on  the  PLCs. 

In  September  2007,  DHS  demonstrated  the  feasibility  of  customized  targeted 

attacks  on  ICS  systems  with  the  Aurora  exploit  [35].  Aurora  damaged  rotating  electrical 

equipment  with  multiple  torque  shocks  by  opening  a  breaker  then  closing  a  generator 

back  into  the  power  system  out  of  phase  [36].  This  electrical  attack  could  be  conducted 

locally  or  remotely  using  unauthorized  access  to  conduct  man-in-the-middle  or  address 

resolution  protocol  (ARP)  cache  poisoning  to  inject  breaker  trip  commands.  Although 
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several  mitigation  teehniques  eurrently  exist  [37],  Aurora  proved  empirieally  the  ability 
to  attack  a  physical  device  via  the  Internet  and  underscored  the  critical  need  to  identify 
malicious  access  to  ICS  assets. 

In  June  2010,  a  piece  of  malware  was  uncovered  by  researchers  from  Belorussian 
security  firm  VirusBlokAda  that  would  eventually  become  known  as  the  Stuxnet  worm. 
Stuxnet  exploited  four  zero-day  vulnerabilities  and  used  two  compromised  digital 
certificates.  The  malware  did  not  target  the  PLCs  and  field  devices  directly;  it  exploited 
high-level  application  software  on  the  controlling  Windows-based  systems  with  the 
ability  to  program  PLCs  [4].  The  PLC  rootkit  code  resided  on  and  was  executed  on  the 
engineering  workstation  and  not  the  PLC  itself  [38].  The  worm  affected  Windows  2000, 
Windows  XP,  Windows  Server  2003,  Windows  Vista,  and  Windows  7  workstations.  In 
addition  to  probable  initial  universal  serial  bus  (USB)  device  infection,  Stuxnet  spread 
over  the  LAN  using  network  shares  [39].  Researchers  have  concluded  that  Stuxnet  code 
remained  idle  for  an  average  of  12.8  days  after  initial  infection  before  executing,  then 
persisted  quietly  for  26.6  days  between  subsequent  executions  [4]. 

Although  data  deletion  attacks  have  existed  since  at  least  1998,  when  the  CIH 
virus  was  created  to  overwrite  a  portion  of  the  hard  disk  as  well  as  the  computer’s  flash 
read-only  memory  (ROM)  [40],  the  recent  resurgence  of  these  attacks  has  renewed 
concern  over  this  style  of  attack.  In  August  2012,  the  Shamoon  data  deletion  attack 
targeted  the  world’s  largest  oil  company,  Saudi  Aramco.  ICS-CERT  claims  that  a 
destructive  attack  similar  to  Shamoon  could  just  as  easily  have  occurred  on  ICS  networks 
[41]  and  some  reports  state  that  data  deletion  attacks  were  executed  against  Iran’s  Oil 
Ministry  control  systems  in  Kharg  Island  around  the  same  timeframe  as  the  Saudi 
Aramco  sabotage  [42].  Shamoon  executes  a  copy  of  itself  as  a  scheduled  job,  entrenches 
itself  as  a  service,  drops  a  malicious  driver  which  is  loaded  and  executed  to  obtain  disk 
access,  then  overwrites  disk  data  starting  with  user  data  then  system  data  and  eventually 
the  system’s  master  boot  record  (MBR).  The  cyber  attack  effectively  wiped  30,000 
computers  from  Aramco,  disabled  some  of  its  internal  networks  for  weeks,  and  was  soon 
spread  to  other  oil  and  gas  firms,  such  as  RasGas  [43]. 
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In  2014,  the  first  publicly-released  prototype  PLC  rootkit  with  multiple  payloads 
that  can  remain  embedded  in  the  device’s  firmware  appeared  [44],  The  rootkit  conducts 
denial  of  service  attacks  and  overwrites  flash  memory  to  make  the  attack  persistent 
through  device  reboots.  Planned  future  efforts  by  the  researchers  include  the  ability  to 
distribute  the  malicious  rootkit  to  other  PLCs  using  available  communication  channels 
and  adding  telemetry  data  tampering  functionality  to  the  rootkit  [44], 

2,  Commodity  Malware  in  the  ICS  Domain 

Malicious  activity  in  the  ICS  domain  can  include  the  intentional  or  accidental  use 
or  re-purposing  of  known  malicious  software  such  as  crimeware  or  banking  Trojans. 
Even  though  these  exploits  and  malicious  software  were  not  written  to  target  ICS 
systems,  commodity  malware  from  traditional  systems  introduces  instability  into  an 
already  volatile  ICS  domain,  with  unknown  effects  on  the  operation  cyber-physical 
systems.  Because  antivirus  signatures  exist  for  much  broad- audience  malware,  the 
detection  rate  is  high  and  there  is  more  publicly  available  technical  information  than  for 
the  other  categories.  Often,  regulated  critical  infrastructure  entities  must  report  these 
incidents  and  thus  more  are  disclosed.  We  describe  some  representative  examples. 

In  January  2003,  the  Nuclear  Regulatory  Commission  (NRC)  confirmed  that  the 
Microsoft  Structured  Query  Language  (SQL)  Server  worm  known  as  the  Slammer  worm 
infected  the  Davis-Besse  nuclear  power  plant  in  Oak  Harbor,  Ohio.  The  worm  spread 
indiscriminately  and  infected  one  of  the  plant’s  contractor  computers,  which 
consequently  bridged  a  fiber  optic  connection  from  their  computer  to  the  internal 
SC  AD  A  network  of  the  plant  [45],  bypassing  a  firewall  that  was  configured  to  block  the 
specific  user  datagram  protocol  (UDP)  port  used  for  propagation.  The  worm  infected  an 
unpatched  server  in  the  ICS  network  and  caused  enough  network  congestion  to  shut 
down  the  Safety  Parameter  Display  System  for  nearly  five  hours.  Reporting  indicates  that 
the  Slammer  worm  also  impacted  other  electricity  sector  systems  [45]. 

In  August  2003,  a  variant  of  the  Sobig  worm  was  introduced  into  the  CSX 
Railway  headquarters  in  Jacksonville,  Llorida.  This  commodity  malware  installed 
applications  and  created  backdoors  while  continuing  to  spread  by  infecting  e-mail 
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attachments.  Although  not  specifically  designed  for  or  targeting  the  railway  systems,  the 
worm  propagated  into  the  control  center  causing  loss  of  signaling,  dispatch,  and  other 
related  systems.  Reporting  indicates  that  Amtrak  trains  in  the  area  were  also  affected  by 
the  malware’s  introduction  into  CSX  Railway’s  systems.  The  end  result  was  multiple 
train  delays  and  expensive  clean-up  costs. 

In  August  2005,  thirteen  United  States  automobile  manufacturing  plants  operated 
by  Daimler  Chrysler  were  shut  down  when  the  Zotob  worm  was  introduced  into  the 
control  network.  This  commodity  malware  was  introduced  to  the  ICS  environment 
despite  the  existence  of  professionally-installed  firewalls  and  studies  of  the  incident 
concluded  that  a  secondary  pathway  bridged  network  zones  [46].  The  plants’  ICS 
networks  remained  offline  for  almost  an  hour  and  additional  plants  in  Illinois,  Indiana, 
Wisconsin,  Ohio,  Delaware,  and  Michigan  were  also  forced  down.  The  outage  forced 
approximately  50,000  assembly  line  workers  to  cease  work  and  cost  Daimler  Chrysler  an 
estimated  $14  million  in  losses  [46]. 

In  October  2006,  an  attacker  penetrated  the  network  at  a  Harrisburg, 
Pennsylvania,  water  filtering  facility  [47].  The  hacker  compromised  an  employee’s 
business  laptop  over  the  Internet  and  then  introduced  malware  into  the  water  facility’s 
SC  ADA  system  using  the  compromised  laptop’s  trusted  remote  access  mechanism.  The 
intrusion  was  discovered  prior  to  damage  occurring  and  thus  the  systems  remained  stable 
and  operating. 

In  August  2008,  viruses  intended  to  steal  passwords  and  send  them  to  a  remote 
server  jumped  a  significant  physical  air  gap  and  infected  laptops  inside  the  International 
Space  Station.  Although  the  impacts  were  minimal,  the  virus  did  make  it  onto  more  than 
one  laptop,  suggesting  that  was  spread  using  internal  networking  or  portable  media. 

In  early  2012,  ICS-CERT  provided  on-site  support  at  a  power  generation  facility 
where  common  malware  had  been  discovered  in  the  ICS  environment  and  was  likely 
introduced  accidentally  by  an  employee’s  use  of  a  USB  drive  to  back  up  control 
configurations.  Several  machines  were  likely  affected  by  the  incident,  including  two 
Windows-based  engineering  workstations,  both  critical  to  the  operation  of  the  control 


22 


environment.  Aeeording  to  ICS-CERT,  “these  workstations  had  no  baekups  and  an 
ineffeetive  or  failed  eleanup  would  have  signifieantly  impaired  their  operations”  [48]. 

In  Oetober  2012,  ICS-CERT  responded  to  an  ineident  in  whieh  the  Mariposa 
seamming  botnet  was  diseovered  on  ten  eomputers  within  a  power  eompany’s  turbine- 
eontrol  system.  This  ineident  was  eaused  when  a  teehnieian  used  a  USB  drive  to  upload 
software  updates  during  a  seheduled  equipment  upgrade  outage.  Quarantine  and 
restoration  proeedures  at  the  eompany  resulted  in  downtime  for  the  impaeted  systems  and 
a  delay  of  the  plant  restart  for  three  weeks  [48]. 

In  January  2014,  malieious  aetors  modified  the  widely-available  “GhOst  RAT” 
Trojan  and  infeeted  the  Monju  fast-breed  nuelear  reaetor  in  Tsuruga,  Eukui  Prefeeture 
[49].  RAT  stands  for  remote  administration  tool  and  also  remote  aeeess  Trojan. 
Commodity  RATs  like  GhOst  RAT  and  Shady  RAT  have  been  used  for  intrusions  into 
eritieal  infrastrueture  networks  sinee  at  least  2009  [42].  Using  a  webshell  hidden  within 
an  image  on  a  popular  media  player’s  update  server,  GhOst  RAT  was  downloaded  to  one 
of  the  eight  eomputers  in  the  Monju  reaetor’s  eontrol  room  [50].  Analysis  of  the  ineident 
has  revealed  that  the  webshell  had  been  in  plaee  sinee  2011  and  the  modified  GhOst  RAT 
malware  was  eompiled  in  2013,  indieating  that  the  adversary  was  patient  and  planned  out 
the  intrusion.  The  webshell  redireeted  users  at  the  Monju  plant  to  another  web  server 
where  a  self-extraeting  eompressed  arehive  that  eontained  multiple  malieious  modules 
was  downloaded  and  various  persistenee  meehanisms  were  established.  Outbound 
eommand  and  eontrol  traffie  was  eventually  manually  observed  by  on-duty  personnel 
while  filing  paperwork  on  the  system,  weeks  after  initial  infeetion.  The  harvesting 
module  of  this  malware  allowed  the  eompromised  maehine  to  be  aeeessed  more  than 
thirty  times  in  a  five-day  period  and  resulted  in  the  pilfering  of  over  42,000  e-mails, 
meeting  materials,  Monju  re-organization  doeuments,  and  staff  training  reeords  stored  on 
the  maehine  [49].  This  intrusion  only  added  to  the  troubled  history  of  the  Monju  nuelear 
reaetor  and,  following  the  intrusion,  the  plant  was  seleeted  for  deeommissioning. 
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3. 


Unauthorized  Persistent  ICS  Network  Access 


A  third  category  of  ICS  incident  refers  to  any  unauthorized  persistent  access  on 
control  center  systems  or  field  devices  from  another  network  such  as  from  the  corporate 
network  or  the  Internet.  This  is  generally  accomplished  by  subverting  access  controls  for 
built-in  remote  ICS  network  access  solutions,  for  which  malicious  use  is  difficult  to 
detect.  Interestingly,  this  category  is  closest  to  the  official  DHS  definition  of  an  ICS 
threat,  which  is  “persons  who  attempt  unauthorized  access  to  a  control  system  device 
and/or  network  using  a  data  communications  pathway,  either  trusted  internal  users  or 
remote  exploitation  by  persons  unknown  via  the  Internet”  [51].  In  fact,  between  2001  and 
2006,  70%  of  security  incidents  involving  ICS  networks  originated  outside  of  the 
network  [52]. 

In  1992,  a  fired  employee  hacked  into  Chevron’s  systems  in  New  York  and  San 
Jose,  California,  reconfiguring  the  emergency  alert  network  so  that  it  would  crash  [53]. 
The  intrusion  was  not  discovered  until  an  emergency  occurred  at  a  Chevron  refinery  in 
Richmond,  California  and  the  system  could  not  be  used  to  notify  the  adjacent  community 
of  the  release  of  a  noxious  substance.  Network  configuration  details  have  not  been  made 
publicly  available  to  understand  if  and  how  the  emergency  alert  network  was 
interconnected  with  ICS  devices,  but  the  critical  alert  system  was  intended  to  cover 
twenty-two  states  and  several  regions  of  Canada.  The  unauthorized  tampering  resulted  in 
a  ten-hour  system  outage  [53]. 

In  March  1997,  a  Boston  teenager  connected  his  personal  computer  without 
authorization  to  a  dial-up  loop  carrier  system  servicing  the  Worcester  airport  and 
subsequently  sent  a  series  of  commands  that  disabled  it  [54].  The  teenager’s  actions 
altered  the  integrity  of  the  data  and  severed  communication  links  to  an  airport,  forcing  air 
traffic  controllers  to  rely  on  manual  overrides  and  backup  systems  for  six  hours. 
Additionally,  the  unauthorized  access  resulted  in  the  disabling  of  phone  service  to  over 
600  homes  in  the  area  and  affected  the  local  weather  service  and  fire  department.  Most 
notably,  the  teenager’s  access  temporarily  disabled  the  radio  transmitter  that  allowed 
aircraft  to  send  an  electronic  signal  to  activate  runway  lights  on  approach.  Information 

from  the  criminal  case  indicates  that  the  loop  carrier  system  operated  by  the  telephone 
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company  was  accessible  via  modem  to  allow  technicians  to  quickly  change  and  repair 
service  to  customers  from  remote  computers. 

In  April  2000,  Vitek  Boden  wirelessly  operated  sewage  treatment  equipment  at 
the  Marooehy  Waste  Water  faeility  in  Queensland,  Australia  using  a  laptop  in  his  oar  [2], 
Over  the  course  of  three  months,  Vitek  Boden  suocessfully  intruded  into  the  SCADA 
system  46  times,  falsifying  his  network  address,  sending  false  data  and  instructions,  and 
disabling  alarms  that  released  thousands  of  gallons  of  raw  sewage  into  rivers,  parks,  and 
tourist  destinations  [2],  The  series  of  unauthorized  acoesses  resulted  in  residents 
relooating  due  to  the  foul  smell,  the  discoloration  of  the  looal  waterways,  over  $1  million 
in  estimated  oosts,  and  substantial  loss  of  marine  life  [3].  During  the  subsequent 
investigation,  it  was  discovered  that  Boden  previously  worked  as  an  engineer  at  the 
vendor  supplying  Marooehy’ s  ICS  eomponents  and  he  was  attempting  to  obtain  a 
eonsulting  job  to  solve  the  problems  he  was  ereating  [2], 

In  August  2006,  two  Los  Angeles  city  employees  compromised  the  network  that 
controlled  the  city’s  traffic  light  system.  [55].  The  pair  disrupted  traffic  signals  on  the 
signal  control  boxes  at  four  critieal  intersections,  resulting  in  significant  backups  and 
delays.  This  particular  intrusion  was  launched  prior  to  a  labor  protest  by  city  employees. 

In  2009,  a  28-year-old  disgruntled  former  IT  contractor  for  Pacific  Energy 
Resourees  remotely  hacked  into  computerized  eontrols  that  deteeted  leaks  on  off-shore 
oil  platforms  off  the  coast  of  California  [56].  Mario  Azar’s  unauthorized  aeeess  resulted 
in  the  erash  of  telemetry  systems  and  operational  data  unavailability.  The  systems  were 
disabled  for  almost  two  months  before  the  intrusion  was  identified.  This  ICS  safety- 
eomponent  failure  caused  thousands  of  dollars  of  damages  for  Pacific  Energy  Resourees 
but  did  not  trigger  any  leaks  or  physical  consequences. 

In  November  2011,  a  hacker  who  called  himself  “PrOf’  demonstrated  that  by 
using  the  Internet-eonnected  device  enumeration  tool  Shodan,  he  could  access  HMIs 
within  a  South  Houston  water  utility’s  ICS  network  [5].  PrOf  found  that  the  water  utility 
was  running  the  Windows-based  Siemens  Simatic  HMI  software,  a  web-based  dashboard 
for  remote  aeeess  to  their  SCADA  systems,  and  was  connected  directly  to  the  Internet 
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with  only  a  simple  three-character  password  for  protection.  According  to  ICS-CERT,  this 
is  one  of  countless  successful  unauthorized  ICS  network  intrusions  as  a  result  of  the 
exponential  growth  of  Internet-connected  ICS  systems  [48]. 

In  May  2013,  United  States  intelligence  agencies  traced  an  intrusion  into  the 
United  States  Army  Corps  of  Engineers  (USACE)  National  Inventory  of  Dams  (NID)  to 
a  suspicious  IP  address  [57].  The  database  contained  sensitive  details  on  vulnerabilities  of 
the  8,100  major  dams  across  the  United  States.  Although  technical  details  are  not 
available  to  determine  the  interconnection  between  this  database  and  the  ICS  network, 
the  malicious  user  did  maintain  persistent  access  to  the  NID  database  for  at  least  three 
months  [57]. 

In  May  2014,  ICS-CERT  reported  that  a  sophisticated  threat  actor  accessed  a 
company’s  SCADA  server  that  operated  mechanical  equipment.  The  SCADA  server  was 
directly  connected  to  the  Internet  via  a  cellular  data  connection  with  no  firewall  or 
authentication  controls  in  place.  The  intruder  maintained  access  to  the  system  over  an 
extended  period  of  time  and  connected  over  hypertext  transfer  protocol  (HTTP)  and 
SC  ADA-specific  protocols,  although  no  man-in- the-middle  injection  or  system 
manipulation  was  observed  [26].  In  the  same  May  2014  release,  ICS-CERT  also 
disclosed  that  a  public  utility  was  compromised  by  a  malicious  group  that  gained 
unauthorized  access  to  the  utility’s  control  systems.  The  ICS  network  was  configured  for 
remote  access  capabilities  over  the  Internet  with  password-based  authentication  which  the 
attackers  were  able  to  crack  with  brute-forcing  [26].  The  public  release  did  not  include 
any  additional  technical  data  or  statements  regarding  impacts  to  the  utility’s  operations, 
but  the  ICS-CERT  report  notes  that  the  system  owners  were  previously  unaware  of  the 
insecure  configuration  and  that  the  targeted  systems  were  likely  subject  to  prior  intrusion 
activity. 

B,  KEY  POINTS  AND  TRENDS  FROM  ICS  INCIDENTS 

The  studied  intrusions  often  started  within  the  business  network,  or  the  business 
network  was  compromised  as  a  reconnaissance  point  for  follow-up  intrusions  into  the 
ICS  networks.  By  compromising  the  underlying  operating  systems  of  the  workstations 
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hosting  ICS  software,  adversaries  were  able  to  abuse  trust  relationships  between  those 
eompromised  systems’  software  applieations  and  ICS  hardware.  Any  aetion  available  to 
an  ICS  system’s  legitimate  root-level  Linux  user  or  more  eommonly  a  system-level 
Windows  user  may  also  be  performed  by  an  adversary  without  the  use  of  any  ICS- 
speeifie  knowledge  or  exploit  tools. 

In  all  studied  ineidents  involving  intentional  malicious  impacts  to  ICS  systems, 
the  adversary  maintained  a  presence  on  the  target  system  and  had  a  communication 
method  to  communicate  orders.  In  rare  cases,  malicious  access  was  achieved  directly  to 
the  field  devices  over  radio  or  other  connection  method  to  the  endpoints,  however  the 
majority  of  incidents  involved  the  compromise  of  Windows-based  supervisory 
workstations  that  monitor  and  control  field  devices.  Targeted  attacks  and  non-targeted 
commodity  malware  events  also  incorporated  early  rogue  connection  to  the  ICS  network 
and  thus  unauthorized  access  is  a  prerequisite  for  all  incident  categories.  Focused  efforts 
to  identify  attempted  or  achieved  unauthorized  ICS  network  access  may  provide  valuable 
early  indication  of  many  types  of  malicious  activity. 

Because  availability  requirements  often  prohibit  the  restarting  of  supervisory  ICS 
machines,  less  sophisticated  methods  were  required  to  maintain  access  to  the  target 
networks.  Unauthorized  access  generally  persisted  for  a  significant  amount  of  time  in 
these  case  studies  allowing  attack  planning  and  reconnaissance.  The  relatively  simple 
persistent  access  methods  and  lengthy  undetected  malicious  access  to  ICS  networks  were 
not  reliably  identified,  according  to  several  post-incident  analyses,  in  part  due  to  the 
constraints  of  auditing  these  systems  and  concerns  of  introducing  instability  into  the 
environment. 

Technical  indicators  of  anomalous  access  and  ICS  network  compromise  have 
been  extracted  from  these  incidents’  publicly  available  technical  reports.  The  incidents 
used  malicious  methods  to  target  ICS  field  devices  as  well  as  more  traditional  intrusion 
hacking  techniques  like  establishing  external  command  and  control,  scheduling  tasks  and 
modifying  the  registry  to  survive  reboots,  tampering  with  processes  and  services, 
implanting  files  for  future  action,  laterally  moving  within  the  network,  exploiting 

portable  media  devices,  and  abusing  trusted  channels  to  obtain  and  maintain  an  attack 
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position  on  critical-infrastructure  systems.  The  speetrum  of  methods  was  similar  to  that 
of  attaeks  on  eomputer  systems  and  networks  in  general  therefore  existing  effeetive 
identifieation  teehniques  should  be  carefully  adapted  for  use  in  ICS  environments. 
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IV.  METHODOLOGY 


A.  FRAMEWORK  FOR  IDENTIFYING  MALICIOUS  ACTIVITY 

The  design  of  a  technique  to  identify  malicious  ICS  activity  requires  careful 
planning  to  obtain  unique  ICS  data  sources  and  collect  and  analyze  that  data  under  the 
strict  operational  constraints  of  critical  networks. 

1,  Framework  Overview 

The  proposed  malicious  activity  identification  methodology  consists  of  collection, 
analysis,  and  decision  components  for  host-based  and  network-based  ICS  artifacts 
(Figure  3).  The  framework  is  modeled  after  modern  intrusion  detection  techniques 
employed  in  traditional  networks  [13];  however,  these  solutions  are  rarefy  deployed 
correctly  for  ICS  networks  at  critical  infrastructure  sites  [15]  and  most  lack  ICS  protocol 
support  as  well  as  the  signatures  and  behavioral-anomaly  data  necessary  to  identify  ICS 
attacker  tactics  [7].  The  purpose  of  this  technique  is  to  quickly  analyze  an  ICS 
environment’s  systems  and  their  communications,  attempt  to  interpret  abnormalities  with 
a  minimal  required  baseline,  and  isolate  possibly  malicious  activity  and  pathways  on 
critical  networks  in  support  of  security  assessments  and  cyber  incident  response. 


Figure  3.  ICS  Malicious  Activity  Identification  Methodology 
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Although  the  goal  is  the  ereation  of  a  toolkit  that  exeeutes  this  methodology  with 
reliable  high-eonfidenee  output,  the  approaeh  may  need  to  be  eondueted  manually  at  first 
and  eontinually  refined  with  automation  in  mind. 

2.  Framework  Constraints 

ICS  networks  require  that  speeifie  teehnieal  eonstraints  be  understood  and 
adhered  to  for  both  host-based  and  network-based  tools. 

Host-based  tools  are  limited  to  those  installed  and  already  available  on  legaey 
operating  systems.  There  exists  no  eentralized  view  of  system  seeurity  status  to  draw  data 
from,  and  field  deviees  do  not  always  store  logs.  Any  seeurity-monitoring  eommands 
exeeuted  on  these  systems  should  be  run  at  the  lowest  priority  level  to  not  interfere  with 
eritieal  proeesses.  Furthermore,  there  are  temporal  ehallenges  with  ICS  forensie  data 
beeause  proeess  and  state  information  is  often  overwritten  at  a  rate  that  makes 
meaningful  eolleetion  unviable  or  impossible  [6].  The  unique  eonfiguration  and 
availability  requirements  of  these  systems  that  were  explored  in  previous  ehapters 
neeessitate  the  adaptation  of  all  host-based  teehniques  for  eompatibility  and  the  thorough 
testing  of  all  eommands  to  ensure  eritieal  proeesses  will  not  be  interrupted. 

Network-based  teehniques  are  eonstrained  by  the  instability  introdueed  by  aetive 
tools  in  the  environment.  Port  seanning  and  automated  deviee  interrogation  teehniques 
ean  erash  ICS  hardware  by  seanning  too  fast  or  by  sending  null  or  malformed  paekets. 
With  modern  SCADA  serviees  using  internal  Web  servers,  HTTP  GET  and  POST 
requests  ean  eause  aetual  physieal  aetions,  so  even  Web-based  automated  tools  should 
not  be  used  aetively  on  ICS  networks.  Industry  best  praetiees  reeommend,  and  a  few 
examples  in  the  next  seetion  highlight,  the  importanee  of  remaining  entirely  passive  for 
network  traffic  analysis.  That  analysis  must  also  include  the  highly  specialized  and  often 
proprietary  ICS  communication  protocols.  Although  minimal  “noise”  (irrelevant  traffic) 
should  exist  on  ICS  networks,  data  volume  is  an  issue  because  the  amount  of  real-time 
telemetry  data  and  otherwise  unrelated  traffic  can  drown  out  possibly  malicious  packets. 
Furthermore,  ICS  domain  interconnection  methods  and  some  regulator  restrictions  limit 
the  ability  to  tap  critical  networks  at  multiple  locations.  For  these  experiments,  passive 
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network  traffic  captures  were  examined  that  were  collected  using  a  high-capacity  switch 
port  analyzer  (SPAN)  port  on  a  router  or  switch,  a  method  that  should  be  usable  on  any 
ICS  network. 

Several  traditional  penetration  testing  techniques  can  be  translated  for  use  on  ICS 
networks  as  shown  in  Figure  4.  For  example,  instead  of  miming  a  network  port  scan  that 
sends  packets  intended  to  elicit  device  information  that  could  trigger  unexpected  results, 
a  penetration  tester  should  verily  open  ports  on  each  host  in  the  environment  without 
generating  network  traffic.  This  thesis  explores  eliciting  considerably  more  detailed 
incident  response  and  forensic  data  from  these  systems  while  following  similar 
constraints. 


Activity 

Usual  Actions  for  IT 

Preferred  Actions  for 
SCADA 

Identification  of  hosts, 
nodes,  and  networks 

Ping  Sweep  (e.g.  nmap) 

1 .  Examine  CAM  tables 
on  switches. 

2.  Examine  router  config 
flies  or  route  tables. 

3.  Physical  verification 
(chasing  wires). 

4.  Passive  listening  or  IDS 
(e.g.  snort)  on  network. 

Identification  of  services 

Port  Scan  (e.g.  nmap) 

1 .  Local  port  verification 
(e.g.  netstat). 

2.  Port  scan  of  a  duplicate, 
development,  or  test 
system. 

Identification  of 
vulnerabilities  within  a 
sers'ice 

Vulnerability  Scan  (e.g. 
nessus,  ISS,  etc...) 

1 .  Local  banner  grabbing 
with  version  lookup  in 
CVE. 

2.  Scan  of  duplicate, 
development,  or  test 
system. 

Figure  4.  Comparison  of  IT  and  ICS  Penetration  Testing  Actions,  from  [58] 


Accounts  exist  of  security  researchers  ignoring  these  constraints.  For  example,  a 
ping  sweep  was  performed  on  an  active  ICS  network  that  controlled  9-foot  robotic  arms 
and  a  controller  for  an  arm  that  was  in  standby  mode  received  the  ping  sweep  and 
abmptly  swung  around  180  degrees,  luckily  missing  the  person  nearby  [58].  In  another 
instance,  a  ping  sweep  was  performed  to  enumerate  all  hosts  on  an  ICS  network  and  it 
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caused  an  integrated  circuit  fabrication  system  to  fail,  destroying  wafers  worth  $50,000 
[58].  In  a  third  instance,  a  penetration  test  at  a  gas  utility  locked  up  the  SCADA  system 
and  the  utility  was  unable  to  send  gas  through  its  pipelines,  causing  a  loss  of  service  to 
eustomers,  for  four  hours  [58].  Lastly,  in  August  2006,  operators  at  Browns  Ferry 
Nuelear  plant  had  miseonfigured  produets  from  two  different  vendors,  whieh  resulted  in 
exeessive  traffie  on  the  eontrol  network.  This  exeessive  network  traffie  resulted  in  a  high- 
power,  low-flow  eondition  where  the  reeireulating  water  system  eould  not  be  properly 
eooled  [59].  This  event,  whieh  took  the  plant  offline  for  two  days  and  eost  nearly 
$600,000  in  revenue  [59],  demonstrated  the  fragility  of  these  networks  when  exposed  to 
unexpeetedly  heavy  network  traffie.  These  examples  are  proof  of  the  eomplex 
requirements  of  any  seeurity  assessment  or  forensie  aetion  on  eondueted  on  production 
ICS  networks. 

B,  TOOLKIT  SELECTION 

An  ideal  toolkit  requires  the  hand-seleetion  of  the  most  eompatible  host-  and 
network-based  tools  eapable  of  colleeting  and  analyzing  malicious  activity  while  still 
providing  ample  ICS  network  eoverage. 

1.  Host-based  Tools 

To  aid  stability  of  hosts,  an  implementation  should  focus  on  agentless  built-in 
eommands  that  generate  minimal  network  traffie.  Popular  toolkits  sueh  as  Microsoft’s 
Sys internals  require  detailed  eonfiguration  and  have  not  been  thoroughly-tested  on  ICS 
systems,  so  they  are  not  ideal  for  widely-applieable  solutions.  For  host-based  querying 
for  Windows  artifaets,  only  built-in  eommand-line  utilities  should  be  used,  with  an 
emphasis  on  the  Windows  Management  Instrumentation  Command-line  (WMIC)  tool. 
Speeial  eare  should  be  taken  to  ensure  that  the  toolkit  only  attempts  to  query  using 
eommand-line  utilities  that  exist  on  that  hardware’s  possibly-modified  operating  system. 
When  possible,  seripts  should  be  written  for  the  legacy  Microsoft  Disk  Operating  System 
(MS-DOS)  16-bit  command.eom  proeessor  that  preeeded  the  32-  and  64-bit  emd.exe 
found  on  Windows. 
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The  implementation  of  these  scripts  will  be  compliant  with  North  American 
Electric  Reliability  Corporation  (NERC)  Critical  Infrastructure  Protection  (CIP) 
standards’  scanning  policies  [60],  which  ensures  that  the  toolkit  can  be  used  at  not  only 
electrical  utilities  but  at  other  critical  infrastructure  sites  with  ICS  systems  that  are 
subjected  to  the  same  scanning  limitations.  The  tools  are  available  on  several  versions  of 
Windows  operating  systems,  including  most  of  those  found  within  ICS  environments,  as 
illustrated  within  Siemens  Automation  documentation  for  the  use  of  WMIC  to 
troubleshoot  system  patching  (Eigure  5),  representing  a  small  subset  of  this  thesis’ 
planned  usage.  “Product  work-around”  documentation  from  General  Electric  (GE)  shows 
that  WMIC  is  also  supported  on  a  range  of  GE’s  platforms  including  GE’s  Proficy  data 
historian  platform  and  GE’s  Cimplicity  HMI  platform  [61].  References  to  WMIC  can 
be  found  buried  within  most  major  manufacturer’s  supervisory  ICS  system’s 
troubleshooting  documentation.  While  no  documentation  exists  on  the  use  of  these  tools 
for  forensics  or  live  response  on  control  systems,  tailoring  ICS-compliant  queries  can 
identify  unauthorized  access  based  on  real-world  malicious  ICS  activity. 


C:\WlNDOWS\system32\cmd.exe _ 

Microsoft  Windows  XP  [Uersion  5.1.2600] 

<C>  Copyright  1985-2001  Microsoft  Corp. 

C:\>wmic  qfe  list  full  /format :htable  >C:\Tcmp\hotf ixes .htn_ 


Eigure  5.  WMIC  usage  on  Siemens  SIMATIC  WinCC  HMI,  from  [62] 

These  tools  can  run  at  user  privileges  but  offer  the  most  functionality  when  run  as 
an  administrator.  Eor  the  research  and  testing  purposes,  a  new  shared  administrator 
account  with  strong  authentication  was  created  on  every  test  workstation  to  centrally 
query  the  hosts.  Documented  best  practices  recommend  banner  grabbing  as  a  safe  activity 
[58]  and  this  WMIC  node  querying  method  uses  similarly  minimal  network  bandwidth. 
Eor  critical  real-world  networks,  the  toolkit  can  run  the  commands  locally  with  no 
network  traffic  with  the  manual  export  of  results  to  be  collated  on  a  separate  closed 
network. 
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2. 


Network-based  Tools 


The  experiments  used  the  Bro  platform  for  network-based  analysis  [75].  It  is  an 
open-source  network  solution  composed  of  signature  detection,  anomaly  detection,  and  a 
programming  language  designed  to  work  with  network  traffic.  The  signature  detection 
generates  logs  for  which  a  protocol  analyzer  abstracts  details  in  real-time.  Alerts  can  be 
generated  based  on  pre-configured  signature  and  anomaly  notification  rules.  The 
programming  language  defines  the  actions  the  platform  takes  based  on  logic  and 
structured  programming.  Fortuitously,  the  Bro  network  programming  language  was 
updated  in  November  2013  to  support  protocol  parsing  of  the  two  most  popular  ICS 
protocols,  Modbus  and  DNP3.  Bro  can  parse  and  analyze  network  traffic  and  analysis  can 
be  automated  through  the  creation  of  customized  scripts  to  identify  malicious  ICS 
network  activity. 

Figure  6  lists  the  default  protocol  fields  parsed  by  Bro  for  Modbus  and  DNP3. 
Bro’s  ability  to  extract  function  codes  from  ICS  protocol  messages  can  prove  valuable  in 
comparing  network  traffic  against  ICS  protocol  conventions.  The  Modbus  and  DNP3 
analyzers  process  significantly  more  data  than  that  provided  in  Figure  6,  so  custom 
scripts  can  be  written  to  manipulate  register  addresses,  values,  and  additional  payload 
data.  The  created  toolkit  should  scale  in  function  as  future  Bro  ICS  protocol  analyzers  are 
developed. 


ICS  Protocol  Data 

Data 

Type 

Modbus 

Field 

DNP3 

Field 

Message  timestamp 

time 

ts 

ts 

Connection  unique  ID 

string 

uid 

uid 

Connection  orig&resp  host&port 

record 

id 

id 

Message  function  code 

string 

fcrequest 

fc_reply 

Message  failure  exception  code 

string 

exception 

Response  internal  identification  # 

count 

iin 

Figure  6.  Modbus  and  DNP3  Bro  log  fields 
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Additionally,  several  open-source  programming  libraries  such  as  jQuery  and 
jVectorMap  can  be  used  to  aid  in  reporting  and  visual  effects  for  the  network-based 
analysis  toolkit. 

3.  Toolkit  Coverage 

Because  the  most  skilled  adversaries  strive  to  maintain  a  minimal  footprint,  full 
inspection  of  ICS  host  and  network  elements  is  desirable  but  time  and  processing 
constraints,  as  well  as  the  devices’  operational  limitations,  make  it  infeasible  at  this  time. 
Host-based  and  network-based  tools  can  provide  ample  coverage  of  most  ICS  networks, 
while  respecting  constraints  of  device  operation  and  real-time  availability.  Sufficient 
host-based  sampling  has  been  achieved  and  full  population  network-based  data  can  be 
processed  with  this  method,  limited  only  by  the  ICS  network  operator’s  ability  to  collect 
and  store  all  traffic. 

Host-based  tools  should  be  selected  to  provide  very  high  data-driven,  non¬ 
probability  judgment  and  convenience  sampling  rates.  Host-based  tools  should  be  used 
locally  on  compatible  hosts  with  minimal  network  traffic  generated.  Special  logic  should 
be  included  in  the  host-based  scripts  to  ensure  the  stability  and  compatibility  with  various 
legacy  versions  of  the  Microsoft  Windows  OS.  When  possible,  all  scripts  should  be 
generated  with  MS-DOS  and  Windows  4.x  commands,  maximizing  ICS  host  coverage. 

The  network-based  analysis  should  be  completely  passive  by  design  to  not 
interfere  with  reliable  process  control.  Analysis  should  cover  the  spectrum  of  systems 
communicating  over  TCP,  UDP,  and  Internet  control  message  protocol  (ICMP)  protocols 
on  the  ICS  network,  regardless  of  operating  system.  Specifically,  the  network  toolkit 
should  cover  traffic  for  all  hardware  including  field  devices  that  communicate  over 
Modbus  TCP  and  DNP3,  widely  considered  the  two  most-implemented  ICS  protocols  in 
industry  [18].  Support  for  more  ICS  network  protocols  can  be  extended  as  time  permits. 


35 


C.  IDENTIFICATION  OF  ADVERSARY  TACTICS 

This  section  details  proposed  technical  identification  techniques  for  adversary 
tactics  as  observed  in  the  malicious  ICS  activity  case  studies.  When  applicable,  host  and 
network  script  excerpts  are  provided  to  illustrate  usage. 

1.  ICS  Field  Device  Anomalous  Operations 

In  several  cases  examined,  the  only  indication  of  malicious  access  was  the 
anomalous  operation  of  the  ICS  field  devices.  Stuxnet  resided  on  the  WinCC  HMI  and, 
using  a  known  hardcoded  password,  modified  rotating  motor  spinning  frequencies  in 
Siemens  S7-315  programmable  logic  controllers  and  valve  settings  in  S7-417s.  This 
suggests  that  cleartext  protocol  data  should  be  examined  and  ICS  protocol  datagrams 
should  be  inspected  for  known  default  passwords  and  other  vulnerabilities.  Obtaining  and 
validating  upper  and  lower  field  device  register-boundary  values  against  site-specific 
expectations  and  equipment-tolerance  values  may  help  identify  overclocked  or 
maliciously  manipulated  devices.  ICS  protocol  operations  should  also  be  used  to 
automatically  create  a  catalog  of  devices,  such  as  Modbus  function  code  0x11  and  0x2B 
that  query  for  device  details;  responses  to  these  packets  can  be  passively  inspected  for 
field  device  characteristics.  To  extract  high  and  low  values,  the  maximum  and  minimum 
values  for  each  device  register  should  be  stored  and  checked  upon  single  register  request, 
single  register  response,  and  multiple  register  request  events.  One  of  this  study’s  Bro 
scripts  can  access  and  catalog  these  register-boundary  values  from  both  historical  and 
real-time  network  traffic.  The  script  should  capture  register-boundary  values  from 
replayed  historical  samples  then  create  expected  register  limits  with  which  to  monitor 
real-time  traffic,  which  requires  fewer  calculations  and  events.  Because  the  ICS  network 
protocol  data  is  passively  ingested  on  network  infrastructure  SPAN  ports,  and  Bro  is 
designed  to  distribute  and  manage  loads  [75],  no  network  latency  will  be  introduced 
when  running  any  network-based  scripts  on  historical  or  real-time  traffic.  If  too  much 
network  traffic  is  aggregated,  packets  will  merely  be  dropped  on  the  monitored  SPAN 
port;  both  ICS  process  control  and  network  infrastructure  operations  will  continue 
without  degraded  or  disrupted  performance. 
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The  analysis  of  ICS  field  deviee  register-value  ranges  requires  additional 
information  sueh  as  the  make  and  model  of  equipment  and  what  proeess  it  may  be 
eontrolling.  Organizationally  unique  identifier  (OUI)  bits  should  be  extraeted  from  media 
aeeess  eontrol  (MAC)  addresses  to  assist  in  fingerprinting  deviees  for  both  overall 
awareness  and  for  verifieation  of  speeifie  ICS  hardware  limits.  A  repository  of  ICS- 
speeifie  OUI  vendor  information  and  possible  field  deviee  funetion  has  been  researehed 
and  ereated  for  this  thesis  for  offline  eomponent  identifieation.  Unfortunately,  Bro 
abstraets  network  traffie  at  an  early  phase  of  analysis  and  does  not  eurrently  allow  the 
extraction  of  MAC  addresses  from  ICS  protocol  communication. 

To  assist,  dynamic  host  configuration  protocol  (DHCP)  traffic  within  the  ICS 
network  traffic  should  be  examined  for  physical  addresses,  modifying  an  existing  Bro 
policy  and  further  extending  it  with  ARP,  DNP3,  and  Modbus  protocol  analysis. 
Additionally,  the  host-based  scripts  should  query  each  system  to  extract  MAC  addresses 
from  their  local  ARP  tables  for  devices  with  which  they  have  communicated,  and  from 
any  in-range  wireless  infrastructure  (Figure  7).  This  method  should  assist  in  monitoring 
for  ARP  poisoning  as  well.  Dual-homed  devices,  those  systems  that  have  multiple 
network  interface  cards,  are  attractive  pivoting  targets  for  malicious  attackers,  so  any 
MAC  address  with  multiple  IP  addresses  should  also  be  identified. 
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echo  MAC  Address  Vendors  from  IPCONFIG:  a  echo  - 

FOR  /F  "Coken3=ll  delims=:  "  %%A  IN  ('IPCONFIG  /ALL  'I  FINC3TP  /R 
■■[''-]  [0-9a-fA-F]  [0-9a-fA-F]-"  ''I  FT'iOST.-  /V  "Tunnel"  ''I  riNDST-  /V 
"00-00-00-00-00-00"')  DO 
set  MACaddress=%%A 
set  Vendor=!MACaddres3:~0,8! 

FIJ'CSTr  /I  tVendor!  vendorMacs.txt 

echo. 

echo  MAC  Address  Vendors  from  ARP  TABLE:  &£  echo  - 

FOR  /F  "tolcens=2"  IN  ('ARP  -a  ''I  /R  "  ["-]  [0-9a-fA-F]  [0-9a-fA-F] -" ' )  DO 

set  MACaddres3=%%A 
set  Vendor=!MACaddress:~0,8' 

FINDSTR  /I  tVendor!  vendorMacs.txt 

echo. 

echo  MAC  Addresses  for  Nearby  Wireless:  tt  echo  - 

FOR  /F  "token3=4"  %%A  IN  ('NETSH  wlan  show  networks  mode''=b33id  "I  '’TNLilr,  /R 
[0-9a-fA-F]  [0-9a-fA-F]  :"')  DO 
set  MACaddre33=%%A 

REM  Reformat  for  search  database  to 

set  Vendor= 'MACaddress : -O , 2 ! - fMACaddress : -3 , 2 ! - !MACaddress : -6 , 2 ! 

FImCSTR  /I  tVendor!  vendorMacs.txt 


Figure  7.  Host-based  passive  ICS  device  enumeration  batch  script 


Since  ICS  traffic  often  should  be  deterministic  (the  operation  of  a  device  should 
be  predictable  since  critical  processes  rely  on  predictable  outputs  for  given  inputs)  the 
network  traffic  can  be  compared  against  ICS  traffic  conventions  to  isolate  tampering  or 
anomalous  behavior.  This  provides  a  stronger  indication  of  malicious  activity  than  on 
traditional  computer  systems.  One  such  convention  is  the  hierarchical  communication 
where  devices  communicate  one-to-many  or  master-to-slave.  Master  and  slave  roles  can 
be  easily  extracted  for  IP-based  protocols  because  the  source  and  destination  addresses 
for  a  specific  message  type  imply  its  role.  Communication  channels  are  defined  by  device 
function  so  an  HMI  should  only  talk  to  a  PLC,  RTU  or  SCADA  I/O  server  and  an  RTU 
should  probably  not  be  sent  non-ICS  protocol  traffic;  these  should  be  constant  because 
devices  are  rarely  added  or  removed  from  the  network.  Another  convention  is  that  ICS 
field  device  communication  is  consistent  and  so,  where  on  traditional  IT  networks  the 
packet  timing  is  influenced  by  human  interaction,  the  ICS  protocol  traffic  generally 
occurs  on  set  polling  intervals. 
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Additional  logic  should  be  added  into  the  Bro  seripts  for  ICS  field  devices  to 
eheck  for  suspicious  functions  and  operations  that  require  an  explieit  interaetive 
eommand  or  reprogramming.  Engineers  and  operators,  when  presented  with  this  data, 
would  know  instantly  if  it  was  a  eommand  that  they  issued  or  if  it  was  a  possible 
unauthorized  injeeted  eommand.  The  Bro  seript  should  inelude  speeific  verbiage 
detailing  the  potential  seeurity  eoneem  of  the  ICS  field  device  eommand  transmitted.  For 
Modbus,  funetion  eode  0x7D  is  issued  to  initiate  firmware  replaeement.  DNP3  eode 
Ox  IB  is  issued  to  delete  a  file  and  eodes  OxOD  and  OxOE  are  issued  to  restart  a  deviee. 
Those  eommands  would  be  instantly  verifiable  if  detected  in  live  ICS  traffie.  Eess  eritieal 
alerts  should  also  be  ineluded  in  the  scripts  but  require  operator  analysis  to  determine  if 
they  are  malieious  or  if  the  ICS  network  is  miseonfigured.  For  instance,  a  Modbus  slave’s 
exception  eode  of  0x03  means  an  illegal  data  value  was  received.  Sinee  we  have 
established  that  Modbus  is  hardware-agnostie  and  not  aware  of  deviee  register  limits,  this 
exeeption  means  that  the  strueture  of  a  query  was  unexpeeted.  Modbus  exeeption  eode 
OxOB  indieates  that  a  target  deviee  failed  to  respond  or  may  not  be  present  on  the 
network.  DNP3  uses  eode  0x21  for  an  authentieation  error  and  0x82  for  an  unsolieited 
response,  both  of  which  should  be  extraeted  by  the  ereated  seripts,  but  it  will  require  an 
operator  to  diseern  whether  the  presenee  of  the  eodes  indicates  malieious  aetivity  or 
deviee  miseonfigurations. 

Comparing  observed  ICS  eommunieation  against  protoeol  eonventions  like  these 
should  allow  the  identifieation  of  malieious  persistenee  without  the  need  for  an  anomaly- 
based  learning  period. 

2,  External  Network  Communication 

The  majority  of  oases  studied  ineluded  some  form  of  malieious  oommunioation  to 
external  networks.  The  high  signal-to-noise  ratio  within  IP-based  ICS  networks  should 
make  the  identification  of  malicious  external  eommunieation  significantly  easier  when 
oompared  to  traditional  enterprise  networks. 

All  versions  of  the  Mariposa  malware  use  oustom  UDP  datagrams  for 
oommunioation,  and  infected  systems  may  beaoon  frequently  and  send  enorypted 


39 


instructions  and  data  across  a  variety  of  UDP  ports.  Hardcoded  domain  names  are  used  in 
most  variants  as  well.  The  Monju  ineident’s  malware  redirected  from  the  eompromised 
streaming  media  serviee’s  website  to  another  malieious  domain  and  used  eommand  and 
eontrol  traffie  over  TCP  port  443.  The  connections  were  established  with  malieious 
server  testqeasd.tk,  so  domain  name  system  (DNS)  lookups  over  TCP  port  53  would  be 
observed  [50].  DNS  eaehe,  eookies,  and  the  hosts  file  ean  be  examined  for  previous 
outbound  attempts  and  may  help  identify  planned  routes  or  dormant  malware.  Web 
browsers  also  store  typed  uniform  resouree  loeators  (URLs)  in  various  forensie  loeations, 
sueh  as  in  HKCU\SOFTWARE\Miorosoft\Internet  Explorer\TypedURLs  for  Internet 
Explorer. 

Sinee  this  thesis  has  argued  that  ICS  networks  should  not  eonneet  to  the  Internet 
direetly,  all  attempted  or  sueeessful  eonneetions  to  external  IP  spaee  may  be  of  eoneern, 
unlike  business  networks.  Eor  instance,  both  a  half-open  TCP  handshake  and  a 
eonneetion  that  has  been  rejeeted  by  an  external  IP  issuing  a  reset  paeket  to  deny  it  would 
not  be  eonsidered  a  eonneetion  in  most  traditional  networks.  Any  external  eonneetion 
attempts  using  ICS  protoeols  are  of  major  eoneern  and  they  should  be  prioritized  for  the 
user  of  the  toolkit  to  review. 

Eigure  8  provides  an  excerpt  from  this  thesis’  Bro  seript  to  categorize  potentially 
malieious  eonneetions  outside  of  the  ICS  network.  The  first  function  abstracts  and 
simplifies  the  state  of  every  TCP  and  UDP  eonneetion  based  on  the  status  of  its  three- 
way  TCP  handshake.  Selecting  an  event-based  trigger  that  captures  both  attempted  and 
sueeessful  eonneetions  in  network  traffie  ean  be  diffieult;  however,  the  seeond  funetion 
shown  in  Eigure  8  should  trigger  when  a  conneetion’s  internal  state  is  about  to  be 
removed  from  memory.  Every  TCP  and  UDP  unique  4-tuple  soeket  pair  should  reliably 
generate  an  event  at  whieh  time  the  network-based  toolkit  should  apply  logie  to  filter  out 
the  original,  meaningful  external  eonneetions  and  their  eonneetion  statuses.  The  bottom 
of  Eigure  8  also  displays  a  portion  of  the  robust  ICS  port  and  protoeol  pairing  that  should 
be  matehed  to  newly-identified  external  eonneetions.  This  ICS  protocol  tagging  should 
provide  helpful  eontext  for  determining  the  security  risk  of  a  particular  connection  since  a 
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previously-unknown  connection  may  be  of  more  immediate  concern  if  it  is 
communicating  over  an  ICS  protocol. 


function  icsState (intState ;  count,  extState : count,  trans:  port);  string  { 
if  (  get_port_tran3port_proto (trans)  =  tcp)  { 
if  (extState  !=  TCP_INACTIVE)  { 

if  (intState  =  TCP_ESTABLISHED  ||  intState  =  ICP_CLOSED)  return  "ESTABLISHED"; 
else  return  "PATHEXISTS" ; 

)  else  if  (intState  !=  TCP_INACTIVE)  return  "ATTEMPTED"; 

)  else  if  (  get_port_tran3port_proto (trans)  =  udp)  ( 
if  (extState  =  DDP_ACTrVE  )  ( 

if  (intState  =  DDP_ACTIVE)  return  "ESTABLISHED"; 
else  return  "PATHEXISTS"; 

)  else  if  (intState  —  DDP_ACTIVE)  return  "ATTEMPTED"; 

}  else  return  "OTHER";  #  TODO:  Add  ICMP  connection  state  logic 

) 

event  connection_3tate_remove (c;  connection)  ( 
local  respIP;  addr  =  cSid$re3p_h; 
if  (  i3_v4_addr (c$id$re3p_h)  )  ( 

if  (  c$id$re3p_h  !in  reservedprivate  &&  c$id$resp_h  !in  reservedlocal  &&  c$id$ 
re3p_h  !in  reservedcast  )  (  #  subnet  sets  based  on  TANA  RFC  1700,  1918,  6890 
local  loc_re3p  =  loolcup_location(cSid$re3p_h)  ; 
if  (icsState  !=  "OTHER")  { 

writeLog(cSid,  ic3_protocol3  [c$id$re3p_p] ,  loolcup_location(c$id$re3p_h) , 
icsState (c$orig$3tate,  c$re3pS3tate,  c$idSre3p_p) ) ; 

addBroMAPnode 0 ;  ♦  fontat  jVectorMap  marker  output,  JSON: : convert ()  strings 

}  1  }  1 

redef  ic3_protocol3  =  (  ♦  Externally  communicating  protocols  of  interest 
[102/tcp]  =  "ICCP", 

[502/tcp]  =  "Modbus_ICP", 

[10e9/tcp]  =  "Fieldbus", 

[1089/udp]  =  "Fieldbus", 


Figure  8.  External  ICS  network  communication  identification  Bro  script  excerpt 


Passive  IP  address  geolocation  can  be  conducted  within  this  same  script  using  the 
offline  MaxMind  GeoIPLite  database  to  plot  attempted  and  established  connections  from 
the  ICS  network  and  the  protocol  used.  The  geolocation  and  plotting  of  external 
connections  on  a  map  should  help  to  determine  approximately  where  anomalous  traffic  is 
bound.  This  method  would  assist  in  distinguishing  innocuous  connections,  like  packets  to 
several  IP-enabled  meters  within  a  utility’s  service  area,  from  nefarious  connections,  like 
recurrent  unauthorized  beaconing  to  a  foreign  country’s  IP  space  over  ICS  protocols.  Bro 
scripts  can  be  written  to  convert  and  store  relevant  connection  attempts  using  JavaScript 
object  notation  (JSON).  Formatting  processes  can  be  conducted  during  Bro’s 
initialization  and  completion  to  directly  write  output  to  a  JSON  file  as  shown  in  Figure  9. 
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This  offline  JSON  data  feed  should  be  eontinuously-updated  while  also  being  projeeted 
onto  a  world  map  using  the  jVectorMap  library. 


var  dataArray  «  [ 

/*  [  {  latLng:  [loolcup_location(c»idSresp_hJlatitude) ,  lookup_locacion(c»id»resp_h»longitude) ]  }, 

name:  JSON:  :convert(c«idSresp_hScity+",  "+c«idSresp_Mregion+", 
ic3_protocol3 (ctid$re3p_p] ,  c$id$orig_h,  ctidtre3p_h, 

"+c$id$re3p_hScountry_code) , 

ICSconnectionState (c,  get_port_cran3port_proco(c*id5re3p_p) )  ] 

[  (  latLng:  [38.778,-77.125],  name:  "Alexandria,  VA,  OS"  ), 
"MODBUS_TCP" ,  "192.168.1.227",  "173.66.46.112", 

"ESTABLISHED"  ], 

(  (latLng:  [37.304199,  -122.094],  name:  "Cupertino,  CA,  OS"), 
"HTIP/SSL",  "192.168.1.13",  "143.127.102.40", 

"ATTEMPTED"  1 

*/ 

Figure  9.  Bro  seript  variable  translation  to  j  VeetorMap  overlay  data 


Not  only  should  external  Internet  aeeess  be  disallowed  but  so  should  several  other 
protoeols  within  the  ICS  network  and  ICS  DMZ  [63].  FTP  and  trivial  file  transfer 
protoeol  (TFTP)  are  eommon  ICS  transfer  protoeols  whieh  use  minimal  processing 
power  that  is  ideal  for  limited  hardware  [15].  Not  only  are  FTP  and  TFP  vulnerable 
services,  since  cleartext  passwords  are  used  for  FTP  and  no  login  is  required  for  TFTP, 
but  they  are  known  to  be  used  by  malicious  actors.  Outbound  FTP  sessions  could  indicate 
malicious  intellectual  property  theft,  so  despite  the  legitimate  use  of  FTP  in  some  ICS 
networks,  any  high-bandwidth  FTP  usage  should  be  identified  [64].  Similarly,  inbound 
Telnet  sessions  from  the  corporate  network,  simple  network  management  protocol 
(SNMP)  versions  1  and  2,  and  any  simple  mail  transfer  protocol  (SMTP)  or  other  e-mail 
traffic  should  be  flagged  [65]. 

Shamoon’s  command  and  control  (C2)  traffic  was  configured  to  beacon  home 
every  two  hours.  The  module  responsible  for  sending  information  about  the  infection  to 
the  attacker  would  send  an  HTTP  GET  request  including  the  domain  name,  number  of 
files  overwritten,  IP  address  of  the  compromised  system,  and  a  random  number  [66].  If 
internal  web  servers  are  being  used  for  SCADA  services,  examining  irregular  HTTP 
properties  like  user  agent  strings  can  help  to  identify  automated  attack  tools  or  manually- 
performed  attacks.  Although  the  logging  of  requests  is  unlikely  to  be  enabled  for  internal 
SCADA  servers  that  use  HTTP  as  a  transport  protocol,  HTTP  traffic  should  be  extracted 
and  characterized  by  the  network-based  toolkit. 
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Depending  on  the  eommunieation  methods  used  between  ICS  deviees,  the 
enumeration  of  IEEE  802.  Ix  wireless  aeeess  points  ean  deteet  rogue  eonneetions.  NIST 
reeommends  that  wireless  aeeess  points  be  loeated  on  an  isolated  network  with  minimal 
eonneetions  to  the  ICS  network  [15].  Current  in-range  serviee  set  identifiers  (SSIDs) 
should  be  enumerated  on  ICS  hosts  sinee  not  only  has  the  existenee  of  in-range  SSIDs 
been  used  for  previous  non-ICS  intrusions  but  in  many  eritieal  environments  there  should 
be  no  wireless  aeeess  points  aeeessible.  Additionally,  data  ean  be  extraeted  using  the 
“netsh  wlan  show  profiles”  eommand  to  indieate  if  a  host  has  previously  eonneeted  to 
any  wireless  aeeess  points.  If  run  as  an  administrator,  the  same  eommand  ean  be  modified 
with  “key=elear”  to  list  eleartext  stored  802.11  wireless  passwords,  a  teehnique  that 
should  be  provided  within  this  researeh’s  seripts  for  administrator  awareness. 

3,  Registry,  Startup,  and  Scheduled  Task  Persistence 

The  Mariposa  hot  kit  found  on  a  turbine  eontrol  network  used  a  registry  startup 
method  that  works  on  all  Windows  versions  from  all  aeeounts,  ineluding  limited  guest 
users,  by  modifying  the  Winlogin  registry  entries  to  enable  the  hot  to  start  at  boot  [67]. 
The  primary  malieious  module  of  the  Monju  plant  ineident  added  entries  to  the 
Windows  registry  in  one  of  two  loeations  depending  on  Windows  version  and 
system  arehiteeture.  Baekdoor  keys  are  plaeed  in  the  default  software  elasses  under 
\InProeServer32\@=expand:”C:\WINDOWS\temp\install.oex”  within  HKEY_USERS  in 
the  registry,  allowing  the  malware  paekage  to  launeh  on  system  startup  [50].  Shamoon 
ereated  the  TrkSvr  serviee  that  starts  itself  with  Windows,  whieh  may  have  been 
identifiable  by  monitoring  the  drivers  set  to  load  during  startup.  Malieious  serviees  ean  be 
eonfigured  to  automatieally  restart  after  speeifie  serviee  failures  using  the  SC  utility,  a 
utility  that  ean  also  be  used  to  query  serviees  for  this  property. 

Comparing  the  hosts’  registries  against  eaeh  other  and  also  against  a  elean 
operating  system  baseline  may  be  aehievable  in  ICS  networks,  unlike  most  networks, 
sinee  signifieantly  less  software  should  be  installed  and  user  behavior  is  restrieted.  In  the 
absenee  of  those  registry  baselines,  several  eommon  loeations  are  used  for  malieious 
registry  persistenee  and  several  speeifie  loeations  were  used  by  the  malware  from  the  ICS 
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case  studies.  These  loeations,  as  well  as  the  “shutdown”  key  value  with  a  dynamic-link 
library  (DLL)  loaded,  should  be  ehecked  in  the  registry. 

The  Mariposa  malware  also  overwrote  the  user’s  Desktop.ini  file  that  initializes 
on  restart,  a  file  often  targeted  for  persistent  aeeess.  Plaeing  binaries  and  bateh  scripts 
within  users’  startup  folders  is  a  simple  but  effective  taetie  for  maintaining  persistent 
aeeess.  In  fact,  during  the  Shamoon  ineident,  attaekers  employed  basie  bateh  scripts  to 
assist  in  data  gathering.  The  relatively  low  number  of  users  and  overall  workstations  in 
the  ICS  environment  should  expedite  the  eheeking  for  these  types  of  plaintext  eommand- 
line  instruetion  files. 

Both  Stuxnet  and  Shamoon  used  the  task  scheduler  to  ereate  and  then  delete  tasks, 
a  proeess  that  should  leave  forensic  artifacts  for  collection.  Shamoon  used  the  task 
seheduler  to  ereate  At.jobs  for  privilege  esealation  to  the  system  aceount  and  also  to 
periodieally  run  eommand-and-eontrol  modules.  The  task  seheduler  was  ultimately  used 
to  exeeute  the  wiper  modules  (system32\dfrag.exe  and  system32\dvdquery.exe)  as  well. 
These  names  were  seleeted  from  a  hard-eoded  list  of  binary  names  to  blend  in  with 
system  files.  The  tasks  for  Shamoon  ineluded  logie  for  exeeution  times,  with  the 
Triggers/TimeTrigger  property  of  the  scheduled  task  as  well  as  Actions/Exec  for  the 
explieit  eommand  execution.  If  a  high  number  of  legitimate  seheduled  tasks  exist  in  the 
environment,  only  those  with  logic  triggers  or  eommand-line  exeeution  need  to  be 
examined  for  suspieious  behavior. 

Teehnieal  ineident  data  ean  reveal  multiple  methods  of  seheduling  Windows  tasks 
for  both  persistenee  and  privilege  esealation.  It  is  neeessary  to  eheek  tasks  seheduled 
using  a  variety  of  serviees,  such  as  AT  jobs,  BITSADMIN  jobs,  and  jobs  assigned  with 
SCHTASKS.  To  further  separate  potentially  malieious  tasks  from  innocuous 
administrative  seheduled  jobs,  tasks  with  executable  actions,  explicit  command  line 
variables,  and  logie  triggers  sueh  as  time  should  be  displayed  prominently  in  the  seript’s 
output. 

Seheduling  a  task  on  supervisory  systems  within  ICS  networks  may  also  produee 
a  log  entry  with  EventID  602  for  older  versions  of  Windows  or  EventID  4698-4702  for 
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Windows  6.0  and  up.  Systems  can  also  be  configured  to  log  Event  ID  567  or  4657  when 
registry  keys  are  modified  so  that  a  service  is  started  at  boot.  Querying  event  logs  can  be 
processor-intensive  and  detailed  logging  is  frequently  unavailable  on  ICS  hosts;  however 
the  scripts  created  for  this  thesis  include  relevant  command-line  search  and  extraction  of 
available  event  log  data. 

4,  Process  Injection  and  Hijacking 

Stuxnet  replaced  a  DLL  file  used  by  Siemens  Step  7  to  communicate  with  PLCs, 
allowing  Stuxnet  to  intercept,  manipulate,  and  replay  ICS  traffic.  While  Stuxnet  was 
sophisticated  malware,  cleaning  the  infection  was  as  simple  as  disabling  or  deleting  two 
signed  driver  entries,  mrxclas.sys  and  mrxnet.sys,  from  the  Windows\System32\ 
directory  [68].  These  files,  dropped  early  on  in  the  infection,  are  registered  to  start 
when  the  system  boots.  Although  it  is  difficult  to  check  for  changed  file  listings  across  all 
hosts  in  the  business  network,  it  may  be  feasible  to  monitor  specific  directories  on  ICS 
systems  for  malicious  libraries  intended  for  DLL  search-order  hijacking.  Anomalous 
characteristics  should  be  recorded  for  running  processes  and  services,  like  the  loading  of 
non-Microsoft  DLLs  into  services.exe,  lsass.exe,  or  explorer.exe  or  processes  launched 
from  user  and  temp  directories.  Determining  the  running  processes  that  are  not  executing 
from  the  operating  system  directory  may  be  a  valuable  script  for  live  production  ICS 
systems. 

An  excerpt  from  this  thesis’  initial  ICS  running  process  attribute  whitelist  is 
shown  in  Ligure  10.  Using  the  command-line,  WMIC  queries  should  be  structured  to  list 
all  processes  and  services  whose  name  or  display  name  is  not  equal  to  those  listed  within 
an  external  lightweight  file.  Results  from  all  hosts  should  be  counted  by  unique  process 
and  output  to  a  hypertext  markup  language  (HTML)-formatted  table. 
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Figure  10.  Excerpt  from  this  thesis’  ICS  process  and  service  attribute  whitelist 


The  commodity  malware  used  for  the  Monju  reactor  compromise  used  malicious 
implant  code  run  by  non-malicious  trusted  code,  similar  to  DLL  search-order  hijacking 
with  a  signed  executable  [50].  Because  the  streaming  media  player  service  downloaded  a 
malicious  update,  cleverly  named  as  the  setup  for  the  beta  version  of  the  player,  the 
malicious  modules  all  ran  at  the  same  privilege  as  the  legitimate  streaming  media  player 
process. 

Special  attention  should  be  paid  to  limit  processes  that  are  installed  and  listening 
on  multiple  boxes  which,  if  compromised,  provide  access  to  multiple  hosts  [63].  With 
fewer  systems  and  processes  running  than  traditional  IT  networks,  it  should  be  possible  to 
identify  anomalous  processes  running  in  control  centers  using  WMIC  with  the  /node 
switch  for  remote  hosts  and  a  well-structured  central  query. 

5,  File  System  Sabotage 

The  malware  used  at  the  Monju  plant  hid  itself  with  double  file  extensions, 

padding  .dll  files  as  .tmp  and  .pdf  files  [50].  Running  a  file-type  classifier  such  as  the 
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Linux  “file”  command  or  detecting  extension  mismatches  in  system  or  temp  directories 
can  help  to  identify  previously-unknown  malicious  tiles.  Not  only  can  analysis  be 
conducted  on  the  hosts  for  these  extension  mismatches,  but  also  within  the  network 
traffic  captures. 

While  rare  tile  extensions  may  be  more  common  within  ICS  networks  due  to  the 
systems’  specialized  function  and  regular  dependency  on  proprietary  software,  isolating 
non-standard  software  associations  may  help  identify  tile  types  that  have  been  registered 
to  start  with  malicious  programs.  Additionally,  the  Windows  registry  debugger  value  can 
be  set  to  hijack  tile  associations  to  execute  them  with  malicious  software.  This  is 
accomplished  by  setting  the  debugger  string  value  within  the  registry’s  image  tile 
execution  options  for  a  given  executable  and  can  be  achieved  with  a  single  command. 

The  Mariposa  malware,  a  variant  of  which  has  been  found  on  ICS  networks,  and 
several  other  well-known  malicious  programs  copy  binaries  into  the  C:\RECYCLER 
folder.  Specifically,  Mariposa  copies  a  file  called  dllrun32.exe,  but  finding  any  non- 
deleted  files  in  the  RECYCEER  or  newer  $Recycle.Bin  directories  may  be  of  interest  in 
assessing  the  security  of  the  ICS  domain.  All  files  in  C:\$Recycle.Bin\(SID)\*  should 
have  filenames  that  start  with  a  The  recycling  bin  may  also  contain  interesting 
recently-deleted  data,  so  it  should  be  checked  for  .exe,  .rar.,  .zip,  and  .txt  extensions  to 
consider  files  for  forensic  recovery. 

Although  compressed  archives  may  be  common  at  some  critical  sites  for 
packaging  large  amounts  of  data  to  move  between  segmented  networks,  the  presence  of 
anomalous  compressed  archives  on  ICS  file  systems  has  been  linked  to  malicious  activity 
and  thus  is  included  in  this  method.  The  logs  of  antivirus  and  other  system  scanning 
solutions  may  include  historical  information  on  encrypted  compressed  archives  that  could 
not  be  opened  and  inspected  at  a  previous  time,  even  if  those  archives  no  longer  exist. 

During  the  Monju  reactor’s  infection,  a  self-extracting  compressed  .rar  archive 
was  used  to  deliver  components  [50].  Shamoon  used  compressed  archives  on  the  network 
to  package  reconnaissance  data  including  hostnames  and  IP  addresses  (hostnames.rar, 
ips.rar)  and  also  to  deliver  malicious  tools  like  Windows  credential  editor  (wc.zip. 
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w64.zip).  Even  limited  ICS  hardware  ean  run  the  WMIC  eommand  with 
path=eim_datafde  and  when  passed  a  WMIC  query  language  (WQL)  “where”  elause  for 
the  desired  extensions  should  be  able  to  list  all  eompressed  arehives  on  Windows  hosts  in 
the  environment. 

Automated  file  listing  and  property  eheeks  ean  also  inelude  the  auditing  of  eertain 
applieations  like  Windows  Stieky  Keys  (sethe.exe)  and  Windows  Utility  Manager 
(utilman.exe)  that  are  often  replaeed  for  privilege  esealation  and  malieious  persistenee. 
These  files  generally  require  loeal  aeeess  to  modify  but  they  are  launehed  with  speeifie 
keyboard  sequenees  with  administrator  privileges.  Another  file  listing  triek  that  malieious 
users  ean  exploit  is  the  way  Windows  handles  unquoted  exeeutables.  If  paths  inelude 
spaees,  sueh  as  ‘C:\Program  Files\’  or  ‘C:\Doouments  and  Settmgs\,’  and  referenees  to 
these  loeations  are  missing  the  quotes  around  the  full  path,  Windows  will  separate  file 
items  at  the  spaees.  For  the  examples  provided,  C:\Program.exe  and  C:\Doeuments.exe 
would  be  exeeuted  if  those  fdes  existed  and  referenees  were  missing  fully-quoted  paths. 
Auditing  for  unquoted  exeeutable  usage  ean  be  eondueted  quiekly  with  few  false 
positives  expeeted. 

6.  Internal  Network  Lateral  Movement 

Adversaries  attempt  to  transition  through  internal  networks  to  gain  eontrol  or 
aeeess  on  a  system  inside  the  target  seeurity  zone  from  a  network  presenee  on  a  lesser 
seeurity  zone.  Stuxnet  aeeomplished  this  lateral  movement  by  propagating  over  network 
shares,  exploiting  a  printer-spool  vulnerability,  and  through  Windows  remote  proeedure 
eall  (RPC)  eommands  [39].  Stuxnet  enumerated  user  aeeounts  on  the  eomputer  and 
attempted  to  explieitly  login  with  the  user’s  eredential  token  to  all  network  resourees  and 
exeeute  itself  on  remote  shares.  If  Stuxnet  determined  that  the  ADMINS  share  was 
aeeessible,  it  repaekaged  the  malieious  eode  with  the  latest  eonfiguration  data  bloek  and 
eopied  itself  to  the  share  with  a  .tmp  filename.  A  network  job  was  then  seheduled  to 
exeeute  the  file  two  minutes  later.  Shamoon  also  eopied  itself  to  aeeessible  network 
shares  then,  if  sueeessful,  exeeuted  itself  remotely.  Network  shares  and  shared  printers 


48 


make  attractive  targets  to  span  network  zones  so  the  toolkit  should  attempt  to  identify 
these  pathways  on  ICS  networks. 

A  module  of  GhOst  RAT  used  at  the  Monju  plant  allowed  for  peer-to-peer 
communications  to  further  propagate  through  the  target  network  [50],  so  processes 
listening  or  established  internally  on  unexpected  ports  may  identify  malicious 
infrastructure. 

Internal  lateral  movement  can  also  appear  as  many  flows  from  a  single 
workstation  to  others  that  normally  do  not  communicate,  especially  over  ports  139  and 
445.  If  logs  are  available.  Event  ID  552  in  the  Security  event  log  may  help  identify 
accounts  and  systems  used  to  conduct  this  activity,  and  these  can  then  be  fdtered  to 
isolate  malicious  behavior  and  lateral  movement. 

7.  Portable  Media  Exploitation 

The  creators  of  Stuxnet  probably  used  portable  media  devices  for  the  initial 
infection  vector,  due  to  the  Microsoft  Windows  shortcut  ‘LNK/PIF’  automatic  file 
execution  vulnerability  [39],  allowing  auto-execution  from  USB  drives.  Also,  the  2012 
power  generation  facility  compromise  and  the  turbine  control  network  Mariposa  infection 
published  by  ICS-CERT  resulted  from  employee  and  contractor  usage  of  infected  USB 
drives  [48].  Mariposa  is  known  to  use  the  AutoRun  string  to  spread  via  USB  [67].  ICS 
network  hosts  should  be  audited  to  ensure  that  AutoRun  is  disabled  in  the  registry  for 
portable  media  and  unknown  drives  or  devices.  On  Windows  XP  and  below,  this  setting 
can  be  found  in  the  Policies\Explorer\NoDriveTypeAutoRun  keys  within 
HKEYLOCALMACHINE  and  HKEY_CURRENT_USER  in  the  registry,  so 
extraction  and  interpretation  of  these  values  from  the  command-line  should  be  pursued. 

Furthermore,  most  current  operating  systems  and  even  some  legacy  systems 
provide  queryable  records  of  inserted  portable  media  devices.  Custom  scripts  can  be 
created  to  check  USB  device  connectivity,  make  and  model  of  previous  devices,  and 
other  metadata  to  determine  if  ICS  network  security  policies  are  being  followed  and  if 
malicious  portable  media  exploitation  has  been  performed.  If  the  script  is  executed  across 
multiple  nodes  using  WMIC,  statistical  analysis  can  correlate  unique  identifying  registry 
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values  in  HKLM\SYSTEM\CurrentControlSet\Enum\USBSTOR.  Windows  generates 
and  stores  a  unique  instanee  identifier,  often  the  deviee’s  serial  number,  as  a  subkey  in 
the  registry  and  stores  the  InstallDate,  EastArrivalDate,  and  EastRemovalDate  as 
Property  IDs  0x0064,  0x0066,  and  0x0067  respectively.  The  container  identifier  can  be 
cross-referenced  with  other  variables  elsewhere  in  the  registry  to  determine  if  AutoRun 
was  enabled  and  to  find  the  specific  drive  label.  This  method  should  allow  the  ability  to 
categorize  portable  media  characteristics  such  as  vendor  and  device  model  and  timeline 
each  unique  device’s  usage  across  the  ICS  network. 

8,  User  Activity  Abnormalities  and  Remote  Access  Abuse 

Malicious  attackers  often  create  user  accounts  or  leverage  stolen  credentials  for 
pre-existing  accounts  to  conduct  their  desired  actions.  User  accounts  are  difficult  to 
manage  on  segmented  networks  with  custom  network  architectures  at  each  site  for 
synchronizing  active  directory  and  the  domain  controller  with  the  business  network.  It  is 
expected  that  there  may  be  some  unused  and  incorrectly-configured  accounts  on  ICS 
systems  since  user  access  may  be  more  difficult  to  audit  than  on  other  systems.  With 
comparatively  few  nodes  on  an  ICS  network,  a  full  listing  of  user  accounts  can  quickly 
and  automatically  be  analyzed  for  various  traits  using  command-line  tools.  Host-based 
artifacts,  and  the  corresponding  log  events  if  available,  should  be  analyzed  for  locked-out 
accounts,  users  who  have  never  logged  on,  passwords  that  never  expire,  guest  accounts 
belonging  to  a  group,  blank  administration  passwords,  and  the  creation  of  new  user 
accounts  and  their  naming  convention,  and  simultaneous  sessions  across  multiple 
machines  to  discover  possible  compromised  user  accounts  and  intrusion  pathways. 

One  specific  behavior  that  a  toolkit  should  attempt  to  identify  is  suspicious 
activity  when  users  are  not  at  work.  Many  of  the  studied  incidents  were  either  detected  or 
later  correlated  with  off-hour  access  into  network,  heading  up  the  Shamoon  attack, 
persistence  was  established  and  surveillance  was  regularly  conducted  during  off  hours. 
This  is  a  consistent  trait  for  many  sophisticated  intellectual  property  theft  intrusions  on 
the  business  network  as  well,  with  over  97%  of  a  sophisticated  nation-state’s  1,905 
observed  remote  desktop  protocol  (RDP)  infiltrations  occurring  during  that  country’s 
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standard  business  hours,  which  are  off-hours  for  the  United  States.  For  ICS  networks, 
there  may  be  a  specific  time  window  that  any  organization  would  not  expect  human 
interaction  with  process  control.  Although  certain  situations  may  necessitate  after-hours 
remote  access  to  an  ICS  network,  these  may  be  identified  by  plotting  remote  interactive 
logons  (type  10)  against  time  of  day,  and  likely  the  anomalies  could  be  explained  by  the 
staff 

Identifying  user  behavior  anomalies  has  become  more  challenging  with  modern 
SCADA  engineers’  growing  expectation  of  and  reliance  on  remote  access  to  ICS  systems 
at  other  locations.  The  Worcester  airport  intrusion  was  conducted  by  abusing  dial-up 
remote  access  intended  for  engineers.  Since  some  ICS  networks  still  use  dial-up  modems 
or  wireless  data  for  direct  or  secondary  RTU  access,  these  access  points  may  provide 
valuable  anomalies  from  normal  engineer  access.  The  Shamoon  attackers  enumerated 
RDP  servers  and  attempted  to  use  several  legitimate  tools,  including  PSExec,  Net,  and 
Mstsc  to  exploit  trusted  access  pathways.  ICS-CERT  did  not  release  which  technologies 
specifically  were  used  in  the  May  2014  sophisticated  remote  access  abuse  intrusion  [26], 
however  companion  alerts  issued  reference  RDP,  virtual  network  computing  (VNC),  and 
Sysintemals  tools  including  PsExec.  To  use  any  Sysintemals  tool  you  must  accept 
the  EULA  on  that  system  which  leaves  a  record  within  the  registry,  which  presents 
a  unique  and  quick  way  to  determine  these  tools’  presence  from  the  command-line 
without  searching  the  entire  drive.  To  do  this,  the  value  should  be  checked  for 
Software\Sysintemals\PsExec\EulaAccepted  in  the  registry. 

According  to  a  document  released  by  NERC,  the  Slammer  worm  that  affected  the 
Davis-Besse  nuclear  plant  also  downed  another  electric  utility’s  critical  SCADA  network 
after  moving  from  a  corporate  network,  through  a  remote  computer,  to  a  VPN  connection 
to  the  control  center  LAN  [69].  VPN  logs  can  be  compared  across  hosts  within  the  ICS 
network  and  across  network  zones.  Information  can  be  extracted  from  these  and  other 
related  files  to  identify  anomalous  and  possibly  malicious  usage. 

Separating  legitimate  use  of  trusted  remote  access  tools  from  malicious  use  is 
challenging,  but  several  distinct  properties  of  ICS  networks  may  assist  in  developing  a 

technique  to  reliably  accomplish  this.  Despite  being  highly  valuable  for  analysis,  detailed 
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logging  of  remote  aeeess  events  in  the  ICS  domain  eannot  be  expeeted  at  all  sites,  and 
thus  relevant  forensie  artifaets  for  these  aeeess  methods  must  be  eolleeted  from  various 
loeations  in  the  file  system  using  eompatible  eommand-line  tools. 

D,  CONSOLIDATED  TACTIC  IDENTIFICATION 

If  an  unauthorized  user  had  previous  aeeess  or  eurrently  maintains  malieious 
persistenee  on  an  ICS  network  using  any  of  the  deseribed  adversary  taeties,  forensie 
artifaets  should  be  available  for  eareful  oolleetion  and  analysis. 

Malieious  tampering  with  ICS  field  deviees  eommunieating  over  industrial 
protoeols  may  result  in  anomalous  deviee  operation  and  uneonventional  eommunieation 
patterns.  This  aetivity  should  be  identifiable  with  statistieal  analysis  of  funetion  eodes 
even  without  prior  knowledge  of  the  speeifie  ICS  site  due  to  minimal  network  noise  and 
field  deviee  behavioral  expeetations.  Analysis  should  be  eondueted  primarily  on  ICS 
network  traffie  with  additional  deviee  funetionality  fingerprinting  assistanee  from  host- 
based  queries. 

Network  traffie  should  be  examined  for  attempted  and  established  external 
eonneetions,  sinee  elient-side  attaeks  require  aeeess  from  outside  the  ICS  network. 
External  eonneetion  and  traditional  protoeol  analysis  should  also  reveal  beaeoning 
eommodity  malware  as  well  as  weak  seeurity  praetiees  that  ean  or  have  been  used  as 
adversary  aeeess  ehannels,  with  geoloeation  helping  to  triage  previously-unknown 
eonneetions.  Host-based  artifaets  should  be  used  to  support  historieal  external  aeeess. 

Several  known  loeations  used  for  malieious  persistenee  that  survive  system 
reboots,  sueh  as  speeifie  registry  keys,  system  startup  loeations,  and  seheduled  tasks 
should  be  examined  for  eontent  that  is  unexpeeted  on  eritieal  networks  and  also 
eorrelated  among  deviees  to  identify  software  anomalies.  Host  proeesses  and  their 
imports  should  also  be  examined  to  ensure  only  expeeted  minimal  proeesses  on  eritieal 
nodes.  Popular  file  system  loeations  should  be  earefully  examined  to  identify  data- 
exfiltration  staging  areas  and  the  modified  files  and  extensions  used  for  privilege 
esealation.  The  startup  persistenee,  system  proeess,  and  file  system  artifaets  should  be 
eompared  aeross  hosts  and  analyzed  with  an  ICS-speeifie  attribute  whitelist  and  blaeklist. 
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For  additional  vulnerable  ICS  network  aeeess  veetors,  host-based  artifaets  should 
be  extracted  that  provide  context  on  internal  network  pathways,  portable  media  usage, 
and  remote  access  abuse.  To  increase  analytical  confidence  in  these  findings,  these 
forensic  attributes  should  be  checked  with  on-site  ICS  engineers  and  compared  with 
network  traffic  to  isolate  malicious  behavior  from  observed  anomalies. 
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V.  EXPERIMENTS 


A,  DATASETS 

A  goal  of  this  research  was  to  design  a  reliable  and  effective  ICS  malicious 
activity  identification  toolkit.  This  chapter  provides  an  analysis  of  the  toolkit’s  operation 
and  its  investigative  strengths  and  weaknesses.  Tests  were  conducted  on  a  variety  of  data 
sources,  including  voluntarily-submitted  and  publicly-available  ICS  network  traffic  as 
well  as  researcher-provided  sophisticated  advanced  persistent  threat  (APT)  malware  and 
packet  captures. 

When  sufficient  technical  data  was  unavailable  on  adversary  tactics,  techniques, 
and  procedures  (TTPs),  actions  were  recreated  on  an  array  of  virtual  machines  configured 
with  a  variety  of  operating  systems  and  software  representative  of  a  typical  ICS 
installment.  The  resulting  forensic  evidence  was  used  to  validate  and  enhance  the  host- 
and  network-based  scripts. 

B.  INITIAL  RESULTS 

The  prototype  toolkit  consisted  of  eight  substantial  command-line  utility  scripts 
representing  each  adversary  tactic  researched,  with  several  supporting  batch  scripts  for 
adhering  to  ICS  device  limitations  and  formatting  results.  Although  Bro  is  open-source 
software  that  includes  many  built-in  functions  and  analyzers,  several  custom  Bro  network 
programming-language  scripts  were  written  that  extend  the  coverage  and  capability  of  the 
methodology.  I  have  authored  all  described  host  and  network-forensic  artifact-collection 
scripts  for  each  specific  adversary  tactic  explained.  I  have  matured  and  improved  the 
toolkit’s  scripts  over  time  and  several  excerpts  and  iterations  have  demonstrated  an 
ability  to  collect  forensic  details  within  the  operational  constraints  of  real-world  sensitive 
environments  at  various  critical  infrastructure  sites.  The  most  recent  versions  of  the  file- 
sabotage,  process  injection  and  hijacking,  and  internal  network  lateral  movement  host- 
based  scripts  have  not  been  tested  on  ICS  networks  but  have  performed  as  intended  in  the 
simulated  environment. 
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The  preliminary  findings  from  testing  the  toolkit  ean  be  used  to  assess  the 
methodology’s  performance  and  adherence  to  unique  ICS  constraints.  The  scripts  will  be 
continuously  improved  as  more  data  is  processed  so  that  observed  ICS  activity  and 
potential  pathways  can  be  directly  translated  into  the  toolkit’s  growing  ICS  forensic 
attribute  whitelists  and  blacklists. 

1,  Operation  within  Constraints 

To  operate  within  the  ICS  domain  without  affecting  ongoing  operations,  all  host- 
based  tools  had  to  be  run  at  the  lowest  priority  level  to  allow  process  control  functions  to 
operate  at  full  availability.  This  was  accomplished  by  using  the  built-in  Windows  base 
priority  function  to  force  the  process  and  thread  priority  to  the  lowest  possible  level  [70]. 
Attempts  to  lower  the  host  script’s  priority  from  within  by  referencing  its  own  process  ID 
(PID)  produced  inconsistent  results.  The  most  successful  technique  was  to  launch  all 
host-based  scripts  from  within  a  separate  wrapper  function  that  reliably  set  the  priority 
for  all  spawned  processes.  Set  at  idle  priority,  the  scripts  only  use  free  processing  cycles 
and  do  not  interrupt  any  functions.  Idle  priority  is  normally  used  for  lightweight  software 
such  as  desktop  screensavers  and  applications  that  only  require  periodic  updating.  Within 
the  master  script,  the  user  is  given  the  option  to  increase  this  priority  level  if  faster  results 
are  desired  and  the  system  can  tolerate  it.  All  host-based  scripts  were  successfully 
launched  at  the  lowest  priority  level  and  monitored  to  ensure  that  they  adhered  to  this 
constraint  (Figure  11).  Additionally,  disk  reads  and  writes  were  minimized  through 
similar  process  monitoring  to  safeguard  against  retrieving  too  much  data,  which  could 
overwhelm  a  resource-constrained  ICS  system. 


Af^cations  P 

Processes  |  Services 

1  Performance  |  Networking  |  Users  j 

Image  N... 

Memory  (P... 

Base  Pri  Command  Line 

Desaiption 

* 

i  cmd.exe 

1,104K 

Low  C:\Whdows\system32\cmd.exe  ^  ICSforensics.cmd 

Figure  1 1 . 

Host-based  scripts  running  at  lowest  priority 

The  command-line  utilities  employed  on  the  hosts  were  selected  not  only  based 
on  their  ability  to  produce  reliable  results  for  identifying  each  adversary  tactic,  but  also 
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for  their  support  on  legacy  operating  systems.  The  logic  built  within  the  scripts 
successfully  refrained  from  executing  on  unsupported  versions  by  using  the  VER 
command  and  checking  the  %OS%  and  %COMSPEC%  environment  variables.  The 
WMIC  command,  which  allows  for  detailed  system  management  from  the  command-line, 
can  be  run  with  the  “/node”  option  enabled  to  run  the  host  scripts  across  multiple  systems 
and  centrally  collect  the  data,  while  generating  minimal  network  traffic. 

The  network-based  toolkit  successfully  functioned  without  the  need  for  active 
interaction  with  devices.  This  was  confirmed  by  replaying  historical  network  traffic 
through  the  scripts  to  produce  the  results,  while  running  tcpdump  to  intercept  and  display 
packets  on  the  system  running  the  network-based  scripts.  Although  not  a  requirement,  the 
network-based  toolkit  performed  with  no  delay  while  running  on  a  consumer-grade 
laptop,  demonstrating  that  the  toolkit  is  flexible  enough  to  run  on  any  platform 
compatible  with  Bro.  Custom  Bro  scripts  should  work  under  all  possible  constraints  in 
the  ICS  network  because  they  are  passive  by  design. 

2,  Explanation  of  Host-based  Toolkit  Functions 

Eigure  12  shows  a  use  and  compatibility  matrix  for  this  research’s  host-based 
toolkit,  illustrating  all  commands  used  and  the  operating  system  support  for  each 
command.  The  core  utilities  were  used  in  all  scripts  and  the  individual  commands 
beneath  were  only  needed  for  individual  adversary  tactic  identification  scripts.  Of  note, 
WMIC  is  heavily  used  in  this  toolkit  and  while  it  is  not  included  by  default  in  OS 
versions  prior  to  Windows  XP,  Microsoft  released  a  package  that  allows  it  to  function 
with  legacy  versions  starting  with  Windows  95.  Executing  this  package  on  remote 
machines  requires  some  additional  steps  to  ensure  the  software  is  running  [71].  Any  tools 
that  require  the  installation  of  additional  resource  kit  packages  have  been  noted  on  the 
Eigure  12  [72].  REG  was  used  as  a  quick,  compatible  method  to  query  the  Windows 
registry  for  specific  keys  related  to  the  eight  adversary  tactics  within  all  scripts.  The 
EIND  command  and  the  more  demanding  EINDSTR  command  when  necessary  were 
used  to  incrementally  search  for  values  and  regular  expression  strings  within  the  output 
of  several  other  commands.  WEVTUTIE  was  included  for  its  ability  to  quickly  isolate 
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relevant  event  log  files,  but  its  use  is  optional  since  the  kit  was  designed  with  the 
expectation  that  WEVTUTIL  would  not  be  supported  and  that  the  target  system’s  event 
logs  would  be  insufficient. 
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*Note:  OS  version  requires  installation  of  additional  command-line  utility  package 

Figure  12.  Host-based  command  usage  and  operating  system  compatibility 


The  additional  command-line  utilities  used  only  in  selected  scripts  are  presented 
at  the  bottom  of  Figure  12  in  the  same  order  as  the  corresponding  adversary  tactic  is 
described  in  this  thesis.  Most  utilities  are  run  with  specific,  tested  command  line 
parameters  to  ensure  stability  of  hosts.  The  ICS  field  device  script  used  ARP, 
IPCONFIG,  and  NFTSH  to  extract  MAC  addresses  from  the  local  network,  host 
machine,  and  in-range  wireless  devices,  respectively.  The  external  connection  host-based 
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script  used  IPCONFIG,  NETSTAT,  and  TYPE  to  display  the  DNS  eaehe,  eorrelate  a  host 
proeess  to  network  behavior  observed,  and  to  output  eookies  from  fde  paths  determined 
by  Windows  version. 

While  registry  persistenee  was  eheeked  entirely  with  the  eore  REG  eommand,  the 
startup  portion  of  the  persistenee  seript  employed  TYPE  to  output  eontents  of  eaeh  host’s 
startup  folders  and  SC  to  query  serviees  that  automatieally  restart  or  trigger  based  on 
failure  eonditions.  The  seheduled  task  portion  of  the  persistenee  seript  primarily  used 
WMIC  but  also  applied  the  SCHTASKS  and  BITSADMIN  utilities  to  output  their 
respeetive  seheduled  job  data  to  a  eentral  report  for  identification  of  malicious  dormancy 
and  privilege  esealation  attempts. 

The  proeess  injeetion  and  hijaeking  seript  mainly  used  the  eore  WMIC  tool’s 
querying  language  as  well  as  EIND  to  identify  proeesses  running  from  %TEMP%  or 
%EOCALAPPDATA%  and  to  eompare  proeesses  and  DEE  imports  with  ICS  artifaet 
whitelists  and  blaeklists.  The  proeess  injeetion  seript  also  leveraged  DRIVERQUERY  to 
display  details  about  loaded  unsigned  drivers  as  well  as  a  eonsolidated  list  of  eertifieates 
used  for  the  drivers  that  were  signed  for  easy  analysis. 

The  file  system  sabotage  seript  harnessed  WMIC  to  parse  the  eim_datafile  for 
eompressed  arehives  along  with  TREE  and  DIR  with  speeial  parameters  to  eheek  for 
unquoted  exeeutables,  reeently  ereated  binaries,  and  anomalous  files  in  the  RECYCEER. 
The  file  system  sabotage  seript  also  used  REG  to  query  SOETWARE\Classes  and 
provide  anomalous  debugger  and  image  file  exeeution  keys.  The  file  system  sabotage 
seript  eombined  results  of  ASSOC  and  ETYPE  to  output  file  extensions  and  their  linked 
programs  for  anomalous  extensions  that  do  not  exist  in  the  baseline  operating  system. 

The  internal  lateral  movement  seript  relied  on  WMIC  to  provide  various  shared 
resourees  sueh  as  printers  that  eould  span  network  zones.  The  same  seript  also  used 
NETSTAT  to  eheek  potential  serviees  in  use  and  the  NBTSTAT  tool  to  display  eaehed 
historieal  NetBIOS  eonneetions. 
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The  portable  media  seript  employed  only  the  eore  utilities  to  extract  currently- 
and  previously-connected  portable  devices,  extracting  uniquely-identifying  data  from 
various  registry  entries  to  correlate  their  usage  across  all  hosts.  The  script  also  extracted 
AutoRun  settings  from  various  forensic  locations  to  determine  portable  media 
propagation  risk. 

The  user  behavior  and  remote  access  abuse  script  largely  used  WMIC  and 
WEVTUTIL  (if  available)  to  first  search  for  specific  Event  IDs  if  present,  then  query 
TerminalServices,  RemoteConnectionManager,  and  EocalSessionManager  for  RDP 
artifacts  in  the  absence  of  security  event  log  data.  The  script  was  built  with  input 
parameters  to  whitelist  certain  time  periods  so  as  to  narrow  results  to  specified  anomalous 
off-hours.  The  user  behavior  and  remote  access  abuse  script  also  used  DOSKEY  to 
extract  explicit  command-line  usage  from  memory. 

3.  Concerning  Findings 

The  network-based  scripts  to  passively  identify  external  communication  paths 
from  the  ICS  network  proved  to  be  valuable  and  reliable.  External  connections  from  the 
ICS  network  were  quickly  identified  in  the  studied  data  that  provided  an  understanding  of 
how  the  network  was  configured,  such  as  a  company’s  interconnection  of  an  ICS  network 
with  the  DNS  and  network  time  protocol  (NTP)  servers  on  their  business  network  in 
Eigure  13.  Other  systems  attempted  outbound  connections  to  the  Internet  over  NetBIOS 
and  other  non-ICS  protocol  ports.  In  addition,  pathways  and  network  traffic  fragments 
were  identified  between  several  devices  and  143.127.102.40  in  Cupertino,  CA, 
Symantec’s  EiveUpdate  server  for  virus  definitions;  this  was  further  validated 
when  the  host-based  scripts  extracted  securityresponse.symantec.com  and 
hveupdate.symantecliveupdate.com  from  the  DNS  cache.  A  separate  ICS  network  traffic 
sample  showed  systems  attempting  to  connect  to  guru.avg.com  and  bguru.avg.cz  to 
update  the  Anti-Virus  Guard  (AVG)  antivirus  product.  In  yet  another  sample  data  set,  the 
host-  and  network-based  scripts  quickly  revealed  other  attempts  to  product  update 
websites  for  multiple  ICS  equipment  vendors.  Although  the  inconsistent  attempts  to 
external  networks  in  this  data  were  system  misconfigurations  and  attempted  product 
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updates,  this  analysis  could  have  similarly  identified  beaconing  malware  and  malicious 
external  command  and  control  attempts. 


53  ONS  us  ESTABLISHED 

443  HTTP/SSL  37,304199  -122.094597  Cupertino  CA  US  PATHEXISTS 
123  NTP  us  established 

443  HTTP/SSL  37.304199  -122.094597  Cupertino  CA  US  PATHEXISTS 
NTP  US  ESTABLISHED 

514  unknown  US  ATTEMPTED 

138  unknown  US  ATTEMPTED 

137  NetBIOS  US  ATTEMPTED 


Figure  13.  Identified  communication  paths  from  ICS  network 


The  real-time  visualization  of  attempted  and  established  external  network 
connections  from  the  ICS  domain  provided  value  once  the  automated  offline  analysis  and 
packet  capture  (PCAP)-to-JSON  conversion  scripts  were  completed.  The  jVectorMap 
representation  of  one  real-world  ICS  network’s  external  connection  attempts  is  shown  in 
Figure  14.  When  foreign  IP  addresses  are  included  in  the  JSON  data,  the  map  was 
programmed  to  display  a  world  map. 


<-  CD  http://localhost:80/NPSthesis/BroMAP-beta.html  ^  ^  = 


Figure  14.  External  connections  output  from  this  thesis’  network-based  toolkit 
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HTTP  protocol  analysis  revealed  interesting  eharaeteristies  unique  to  ICS 
networks.  Built-in  Bro  IDS  alerts  were  triggered  beeause  some  traffic  samples  included 
invalid  HTTP  methods.  In  those  cases,  vendors  had  eonfigured  their  product  to  use  a 
[VENDORNAMEJ  POST  method  and  used  HTTP  as  a  convenient  transport  protocol. 
Other  analysis  showed  HTTP  access  to  DEL  files  and  posting  values  directly  as  variables 
(/[vendor]/[vendor].dll?v=update).  These  ICS  device  HTTP  traits  will  be  added  to  the 
growing  whitelist  of  atypical  vendor  implementations  of  traditional  network  protocols. 

USB  drive  insertion  data  is  of  immediate  interest  during  incident  response  and  is 
critical  for  sites  whose  network  security  policies  disallow  portable  media  usage.  Eigure 
15  shows  the  output  of  the  host-based  script  when  run  against  a  test  system  with  many 
USB  drive  insertions.  In  the  prototype  toolkit  there  are  permission  issues  accessing  the 
Properties  subkey  which  contains  dates  that  were  originally  going  to  be  plotted  on  a 
timeline.  This  will  be  fixed  in  the  next  iteration,  and  the  AutoRun  flag  and  the  user  who 
accessed  the  drive  will  also  be  displayed  to  further  assist  security  teams. 


irriCommand  Prompt _ 

Last  Used:  ERROR:  Recess  is  denied. 

-  Patriot  Menory  USB  Device 

Revision:  PMRP 

Serial:  07012C1B16B2F903&0 

ContainerlD:  <905e5ae6-f 649-5b75-8ba4-470495b8441d> 
Last  Used:  ERROR:  Recess  is  denied. 

-  Kingston  DT  101  G2  USB  Device 

Revision:  1.00 

Serial:  001CC0EC34C2FC91370E23F3&0 

ContainerlD:  <89167af 0-22bl-5d49-abf 9-7252665d08a2> 

Last  Used:  ERROR:  Recess  is  denied. 

-  SanDisk  Cruzer  USB  Device 

Revision:  1.20 

Serial:  20054963901B26B12712&0 

ContainerlD:  <f cc908a8-54f 5-5781-a516-8bf 13314085b> 
Last  Used:  ERROR:  Recess  is  denied. 

-  USB2.0  Flash  Disk  USB  Device 

Revision:  2.50 

Serial:  1000000000000732&0 

ContainerlD:  <0075bf e3-4f e7-lle3-96a6-bf e87818eal0> 
Last  Used:  ERROR:  Recess  is  denied. 

-  WD  1600BER  External  USB  Device 


Eigure  15.  USB  insertion  seript  output  on  test  system 


The  above  results  eonstitute  suspicious  data  of  moderate  confidenee.  As  more 
data  is  proeessed,  such  suspicious  findings  can  be  further  separated  into  confirmed 
malieious  findings  in  the  form  of  a  blacklist,  and  moderate-confidence  suspicious 


62 


findings  will  be  used  to  maintain  a  more  fiexible  ICS  behavioral  greylist  that  may  be  only 
indieative  of  eompromise  when  eombined  with  other  observed  aetivity. 


4,  Valuable  Discoveries 

The  initial  version  of  this  toolkit  provided  valuable  eonfiguration  information 
about  a  particular  site  as  well  as  a  more  general  understanding  of  industry  ICS  network 
installments  across  multiple  sets  of  data.  Execution  from  both  the  command-line  of  the 
system  and  on  live  or  historical  network  data  provides  new  observational  capabilities.  For 
instance,  Figure  16  shows  remote  querying  of  an  endpoint  to  identify  connected 
equipment.  This  host-based  script  was  further  improved  to  show  in-range  wireless  access 
points  and  previously  wireless  connections,  if  any. 


00-00-32  Marconi  Pic 
00-00-23  Abb  IndustrialAb 
00-00-23  Abb  IndustrialAb 
00-00-BC  Rockwell  Autonation 
00-00-0C  Cisco 

Press  any  key  to  continue  .  .  . 


Figure  16.  Host  characteristics  extracted  from  command-line  utilities 


Although  host-based  command-line  device  enumeration  will  be  valuable  in 
environments  without  historical  network  traffic  captures,  the  network  scripts  created  in 
Bro  demonstrated  far  more  utility  and  flexibility  in  this  area.  The  network-based 
enumeration  and  correlation  across  several  nodes  is  completely  passive  and  device¬ 
agnostic.  Bro  is  built  around  real-time  generation  of  logs  for  immediate  viewing  and 
alerts,  but  its  language  supports  a  variety  of  outputs  and  data  structures.  Traditional 
stream-based  output  is  supported  in  the  ICS  device-fingerprinting  code  created  for  this 
thesis,  but  the  real  utility  of  the  toolkit  is  the  comprehensive  collection  of  device 
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information  in  scalable  arrays  and  searchable  sets  of  addresses  that  associates  known 
device  information  together  automatically,  without  having  to  further  parse  the  output. 

The  script  pulls  MAC  (physical)  addresses  and  IP  (logical)  addresses  from  ARP 
requests  and  replies,  and  extracts  MACs,  IPs,  and  hostnames  from  DHCP  request, 
inform,  and  discovery  packets.  MAC  addresses  were  passively  observed  for  64%  of 
devices  on  the  LAN  and  100%  of  the  hardware  vendors  were  identified  from  those  MAC 
addresses.  100%  of  the  internal  device  IP  addresses  were  collected,  which  is  to  be 
expected  for  TCP/IP  to  properly  function,  and  none  were  identified  as  dual-homed 
(a  system  having  multiple  network  interface  cards  and  thus  multiple  IP  addresses  per  one 
MAC  address).  Only  12%  of  hostnames  were  observed  in  DHCP  traffic,  likely  due  to  the 
prevalence  of  static  IP  assignments  in  critical  networks.  This  was  a  known  consideration 
during  network  toolkit  creation,  and  so  the  script  includes  an  event  to  trigger  when  a  DNS 
address  (type  A)  reply  is  to  an  internal  IP  address  that  does  not  yet  have  a  known 
associated  hostname  and  for  which  the  corresponding  query’s  subdomain  substring  is 
used  for  the  hostname.  Using  both  DHCP  and  internal  DNS  extraction,  41%  of 
hostnames  were  identified.  These  hostnames  have  been  removed  Figure  17  to  keep  the 
processed  data  private.  According  to  Bro’s  built-in  operating  system  analyzer,  the 
majority  of  operating  systems  in  the  data  were  outdated  and  unsupported  versions  of 
Microsoft  Windows,  validating  the  focus  on  those  systems  with  the  host-based  kit. 
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Vendor 

Count 

% 

Undetermined 

42 

36% 

Hewlett-Packard 

23 

20% 

Dell 

14 

12% 

Vmware 

11 

9% 

Cisco 

8 

7% 

Moxa 

3 

3% 

Cadmus  Computer 

2 

2% 

JK  Micro 

2 

2% 

Bristol  Babcock 

1 

1% 

Fortinet 

1 

1% 

Juniper  Networks 

1 

1% 

Rockwell  Automation 

1 

1% 

Selectron 

1 

1% 

Sixnet 

1 

1% 

Stratus  Computer  De 

1 

1% 

Termtek  Computer  Co 

1 

1% 

Xyplex 

1 

1% 

Yaesu  Musen  Co 

1 

1% 

ZF  Micro 

1 

1% 

System  Total 

116 

Operating  System 

Count 

% 

Undetermined 

71 

61% 

Windows  2000  SP4,  XP  SP  1+ 

34 

29% 

Windows  2000  SP2+,  XP  SPl,  98 

5 

4% 

Windows  XP  SP1+,  2000  SP3 

4 

3% 

Linux  2.6 

2 

2% 

System  Total 

116 

Figure  17.  Summary  of  host  characteristics  extracted  from  network  data 


The  toolkit  extends  the  built-in  fingerprinting  with  some  initial  ICS  protocol  and 
supervisory  device  trait-matching.  For  instance,  based  on  publicly-available  vulnerability 
data  released  for  Wonderware’s  SuiteLink  service  [73],  any  Windows  OS  host  listening 
on  UDP  port  5413  with  connections  to  one  or  more  systems  that  speak  other  ICS 
protocols  is  flagged  by  the  script  as  likely  a  WonderWare  InTouch  FIMI.  This  rough  ICS 
device  cataloging  capability  should  grow  as  more  artifacts  are  ingested  and  processed. 

For  proper  comparison,  24-hour  periods  of  Modbus  and  DNP3  network  traffic 
captures  from  several  critical  infrastructure  sectors  were  analyzed  and  correlated.  One  of 
this  thesis’  Bro  scripts  attempted  to  identify  ICS  protocol  abnormalities  while  another 
script  attempted  to  fingerprint  devices’  specific  roles  in  the  ICS  network  based  on  their 
register  values  and  their  function  and  exception  codes.  The  first  script’s  analysis 
indicated  that  the  ingested  data  did  not  contain  any  protocol  convention  anomalies  or 
suspicious  ICS  function  code  usage,  which  may  indicate  malicious  tampering  or  field 
device  modification.  All  examined  ICS  protocol  traffic  contained  only  register  reads  and 
writes,  which  was  confirmed  through  packet  analysis  outside  of  the  toolkit,  so  the 
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extraction  of  potentially  malicious  activity  based  on  anomalous  protocol  events  was  not 
tested. 

Analyzing  the  same  source  data,  the  second  Bro  script’s  output  did  not  indicate  a 
significant  correlation  between  ICS  protocol  function  code  patterns  or  register-boundary 
values  and  the  field  device’s  role  (e.g.,  HMI,  RTU,  PLC).  This  may  also  be  a  result  of  the 
limited  ICS  device  functions  observed.  Frequency  analysis  of  observed  Modbus  and 
DNP3  function  codes  is  provided  in  Figure  18.  Across  all  samples,  slave  field  devices 
had  an  average  of  5.5  registers.  Register  ranges  varied  significantly  even  on  a  single 
device,  as  in  one  case  a  device’s  register  only  changed  by  within  a  range  of  one  across  a 
24-hour  period  but  the  device’s  other  register  had  a  range  of  65,369.  The  average  low 
boundary  value  for  registers  was  8,594  and  the  average  high  boundary  was  37,248,  but 
the  protocols  are  device-agnostic  so  more  meaningful  anomalies  or  further  indications  of 
malicious  behavior  could  not  be  extracted  from  the  field  devices’  numeric  register-values 
without  knowing  the  specific  processes  they  controlled.  There  are  likely  values  for  field 
device  register  limits  and  metrics  so  alerts  could  be  created  based  on  manually-input 
expectations.  Alternately,  future  iterations  of  the  Bro  script  may  record  register  changes 
as  a  percentage  of  the  current  values.  The  extraction  and  correlation  of  ICS  protocol 
functions  did  however  support  the  assertion  that  ICS  networks  have  a  high  signal-to-noise 
ratio  since  anomalous  function  codes,  exception  codes,  and  malformed  packets  should  be 
significantly  distinguishable  if  present  in  captured  data. 


Protocol  Function  (Code) 

DNP3 

Modbus 

DNP3  Confirm  (0) 

0.02% 

0.00% 

DNP3  Read  (1) 

99.98% 

0.00% 

Modbus  Read  Coils  (1) 

0.00% 

56.26% 

Modbus  Read  Holding  Registers  (3) 

0.00% 

25.90% 

Modbus  Read  Input  Registers  (4) 

0.00% 

13.57% 

Modbus  Write  Single  Coil  (5) 

0.00% 

3.01% 

Modbus  Write  Multiple  Registers  (16) 

0.00% 

1.26% 

Grand  Total 

100.00% 

100.00% 

Figure  18.  ICS  protocol  frequency  analysis  in  dataset 
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C.  TOOLKIT  LIMITATIONS 

This  section  covers  several  weaknesses  in  the  current  version  of  the  methodology, 
including  examples  of  tool-specific  inadequacies  and  a  survey  of  observed  type  I  and 
type  II  errors  in  the  automated  analysis. 

1,  Toolkit  Deficiencies 

One  significant  limitation  of  the  approach  is  its  inability  to  calculate 
cryptographic  hashes  to  verify  authenticity  of  files  due  to  the  absence  of  built-in 
Windows  command-line  hashing.  Since  firmware  is  not  commonly  digitally  signed  for 
ICS  devices,  hashes  should  be  computed  to  compare  against  manufacturer  data  provided 
in  documentation  or  online.  Several  community  tools  are  available  and  Microsoft 
released  a  command-line  hashing  utility  (FCIV.exe)  in  2004,  but  it  is  not  built  into  the 
Windows  OS  versions  examined.  A  PowerShell  script  can  likely  accomplish 
cryptographic  hashing  but  that  limits  compatibility  and  a  scripted  cryptographic  routine 
may  overly  tax  the  limited  resources.  Bro’s  file  framework  is  impressive  and  can 
passively  categorize,  extract,  and  hash  all  files  sent  in  packets,  so  modification  and 
extension  of  Bro’s  file  framework  will  aid  future  firmware  analysis  functionality. 

Several  studied  adversary  attack  techniques  only  produced  accurate  forensic 
artifacts  when  the  same  command-line  utilities  used  by  an  attacker  were  also  used  to 
query  the  host  by  a  responder.  This  suggests  the  need  to  investigate  command-line  utility 
outputs  before  claiming  full  coverage.  For  example,  within  the  task-scheduling 
persistence  identification  script,  simulated  malicious  tasks  for  privilege  escalation  and 
reboot  persistence  created  on  a  Windows  XP  host  with  the  AT  command  were  only 
revealed  by  WMIC,  and  tasks  created  with  SCHTASKS  command  were  only  revealed  by 
querying  SCHTASKS.  On  Windows  Vista  and  up,  a  third  built-in  tool,  BITSADMIN, 
allows  scheduling  of  malicious  tasks  that  are  only  revealed  by  listing  from  BITSADMIN 
and  no  other  tools,  as  illustrated  in  Figure  19. 
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^bits«dnin  ^list  /varbo 


Figure  19.  Three  command-line  tool  queries  with  mutually-exclusive  results 


Although  this  host  script  does  query  each  service  and  identifies  jobs  scheduled 
from  all  methods,  the  output  is  difficult  to  parse  and  thus  a  more  reliable,  central 
location  should  be  used  to  find  these  tasks.  Forensic  locations  such  as 
%windir%\Tasks\SchedLgU.txt  for  Windows  XP  and  below,  and  Microsoft-Windows- 
TaskScheduler%40Operational.evtx  for  Vista  and  above,  may  centrally  store  this  data  for 
all  tools  and  the  scheduled  task  persistence  identification  script  can  extract  data  from 
those  locations. 

Separating  malicious  access  from  trusted  access  was  difficult  in  the  ICS 
environment  despite  the  high  signal-to-noise  ratio.  While  the  toolkit  reliably 
distinguished  automated  device  activity  from  human-initiated  activity,  further  separating 
trusted  user  behavior  from  malicious  attacker  behavior  was  challenging  with  limited 
historical  visibility  and  inconsistent  logging  on  the  systems  analyzed.  Some  promising 
data  was  found  outside  of  command-line  tools  and  parsed  event  logs,  such  as  within  the 
*.rdp  files  on  the  host  and  within  parsed  remote  access  network  traffic,  but  the  initial 
scripts  were  unable  to  automatically  extract  that  data  in  a  meaningful  way  that  applied  to 
multiple  sites. 

Similarly,  malicious  internal  lateral  movement  was  difficult  to  identify  in  this 
environment  due  to  the  number  of  accounts  with  administrative  privileges  and  ICS 
vendors’  pervasive  usage  of  server  message  block  (SMB)  and  NetBIOS  for  access  to 
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shared  resources.  This  should  diminish  with  the  development  of  more  robust  vendor- 
specific  whitelists. 

2.  False  Positive  Errors 

Type  I  errors,  or  false  positives,  occur  when  non-malicious  is  identified  as 
malicious  activity.  The  collection  of  large  datasets  without  honed  capabilities  to  analyze 
the  data  resulted  in  a  high  number  of  false  positives. 

False  positives  were  more  likely  to  occur  when  only  host  or  network  data  was 
available.  Ideally,  toolkits  should  try  to  determine  where  data  went  that  left  a  host  or  what 
process  was  running  that  was  using  a  suspicious  protocol.  In  these  experiments,  some 
host-based  activity  that  was  flagged  as  malicious  was  SCADA  engineers  moving  laptops 
between  network  zones.  It  was  assumed  that  a  host  remained  in  an  environment,  so 
laptops  moved  between  zones  generated  several  suspicious  traits,  such  as  using  many 
fully-qualified  domain  names  that  did  not  actually  represent  external  attempts  from 
within  the  ICS  network.  But  since  connecting  laptops  to  and  from  the  ICS  domain  is  not 
advisable,  these  may  not  be  considered  false  positives. 

Within  one  data  set,  an  indication  of  external  information  exfiltration  over  an  ICS 
protocol,  OASys  DNA,  was  actually  a  Type  I  error  that  resulted  from  incorrect 
programming.  An  external  connection  identification  Bro  script  did  not  originally  include 
255.0.0.3  in  its  whitelist,  a  reserved  multicast  address.  In  another  set  of  data  analyzed,  a 
router  was  configured  to  send  a  response  back  to  attempts  to  connect  out  of  the  ICS 
network.  The  toolkit’s  connection  state  logic  has  since  been  reviewed  and  the  IP  subnet 
whitelist  now  correctly  reflects  all  intended  requests  for  comments  (RFCs)  from  the 
Internet  Engineering  Taskforce  (IETF). 

Within  the  network-based  toolkit  results,  more  activity  was  observed  than 
expected  with  Telnet,  NetBIOS,  SNMP,  and  HTTP  protocols.  Specifically,  many  ICS 
systems  were  observed  using  embedded  HTTP  functionality  for  diagnostics  and 
monitoring.  The  analysis  of  unexpected  HTTP  user-agent  strings  requires  a  more 
comprehensive  understanding  of  the  ICS  software  that  uses  HTTP,  or  the  use  of  filtering 
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by  unique  properties  like  session  duration,  so  that  fewer  anomalies  are  highlighted  to  the 
toolkit’s  user. 


3.  False  Negative  Analysis 

A  Type  II  error,  or  a  false  negative,  is  when  malieious  activity  existed  but  was  not 
detected.  Because  none  of  the  data  included  a  confirmed  compromise,  errors  of  this 
category  were  difficult  to  find.  It  is  important,  to  note  that,  when  searching  for  malicious 
attackers,  the  old  criminal  profiling  aphorism  “the  absence  of  evidence  is  not  the 
evidence  of  absence”  applies. 

One  possible  design  flaw  that  could  lead  to  false  negatives  is  within  the 
geolocation  and  visualization  of  external  connections.  Several  samples  included  external 
attempted  and  established  connections  to  cellular  data  devices.  In  one  instance,  a  SCADA 
server  was  communicating  with  several  dozen  Verizon  devices  with  no  uniform  IP 
address  assignment,  which  could  reflect  sites  replacing  old  direct  dial-up  access  with 
cellular  data  connections.  Geolocation  will  never  be  able  to  place  these  devices 
accurately  on  a  map  since  they  do  not  have  set  locations  and  are  assigned  in  pools  by 
mobile  providers.  Although  separate  online  whois  queries  can  be  conducted  and  collated 
for  some  assistance,  an  adversary  could  blend  into  this  access  pathway  or  maintain 
persistence  by  beaconing  out  of  an  ICS  network  to  an  adversary-owned  mobile  device 
from  the  same  provider. 

Additional  false  negatives  may  have  occurred  in  the  studied  data  due  to  the  lack 
of  built-in  Bro  support  for  ICCP,  OPC,  and  some  other  observed  proprietary  ICS 
protocols.  However,  these  protocols  communicate  in  cleartext  like  many  other  supported 
protocols  (e.g.,  Telnet,  FTP,  MySQL,  and  HTTP),  so  more  work  needs  to  be  done  on  the 
network  toolkit  to  include  the  parsing  of  cleartext  ICS  protocols. 
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VI.  CONCLUSIONS  AND  FUTURE  WORK 


A,  IMPACT 

This  thesis  described  the  need  for  a  forensic  capability  tailored  to  industrial 
control  systems.  It  then  described  and  tested  several  techniques  for  identifying  and 
analyzing  potential  malicious  activity  and  adversary  access  pathways  within  ICS 
networks.  This  resulted  in  the  creation  of  several  tools  to  be  used  for  live  production  ICS 
forensics  and  suggested  some  approaches  to  building  high-confidence  indicators  of 
unauthorized  access.  ICS  networks  do  provide  good  signal-to-noise  ratios  for  behavioral 
anomaly  detection.  The  need  for  tailored  tools  to  identify  potentially  malicious  access 
pathways  has  been  suggested  by  this  work  and  the  feasibility  of  such  tools  has  been 
established  with  the  prototype  host-  and  network-based  toolkit. 

Security  assessment  and  incident  response  teams  can  use  the  approach  to  analyze 
critical  devices  in  an  environment  with  minimal  irrelevant  data,  then  trace  those  findings 
back  into  the  business  network  instead  of  the  other  way  around.  The  tools  should  assist  in 
making  more  confident  intrusion  judgments  when  malicious  activity  is  suspected  in  the 
ICS  environment.  With  the  careful  selection  of  compatible  tools,  and  a  technique  that  is 
passive  and  forensically  cautious,  these  methods  can  be  easily  incorporated  into  many 
organizations’  security  processes. 

B,  FUTURE  WORK 

Due  to  the  relatively  long  life  cycle  of  systems  in  the  ICS  domain  and  the  current 
compatibility  with  Windows  8  systems,  this  technique  is  expected  to  be  functional  for  a 
while  on  Microsoft  operating  systems.  However,  Linux  compatibility  is  needed  and  work 
has  already  begun  on  translating  the  host-based  examination  commands  for  Linux.  The 
WMI  core  application  has  been  ported  to  Linux,  which  presents  a  desirable  host-platform 
shift  since  Bro  runs  on  Linux.  If  the  host  and  network  tools  are  executed  from  the  same 
system,  compatibility  should  expand  and  the  ability  to  cross-reference  host  and  network 
data  should  improve  greatly.  Adding  agentless  querying  of  real-time  operating  systems 
such  as  VxWorks,  Windows  CE,  QNX  and  embedded  Linux  should  also  be  pursued. 
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The  host-based  toolkit  ean  be  run  on  eaeh  host,  transferring  the  HTML-formatted 
output  to  a  write-onee  CD  or  other  proteeted  media.  The  toolkit  ean  also  be  run  on 
multiple  hosts  and  centrally-oolleeted,  but  this  has  only  been  tested  on  small,  elosed 
networks,  and  already  is  generating  signilieant  amounts  of  data.  If  both  host  and  network 
data  ean  be  eentrally  analyzed,  the  data  will  require  improved  struetures  to  ensure 
performanee  is  still  aeeeptable.  This  approaeh  of  eentral  oolleetion  and  analysis  of  host 
and  network  data  allows  additional  features  sueh  as  rootkit  identifieation  to  beeome 
possible  by  eomparing  host  and  network  data.  For  example,  if  a  port  is  eommunieating  in 
network  traffic  but  not  showing  on  a  host’s  netstat  querying,  a  rootkit  may  be  hiding  it 
from  the  Windows  API. 

The  migration  to  a  NoSQL  document  database  for  forensic  data  management 
would  increase  scalability  and  reduce  endpoint  processing  cycles  even  further. 
Alternately,  Bro  may  be  capable  of  ingesting  the  host-based  forensic  output  instead  of  a 
NoSQL  database  and  it  could  be  used  as  a  source-agnostic  state  machine  to  process 
events.  More  interactive  mapping  of  specffically-filtered  data  could  be  valuable. 
Examples  are  the  data-driven  documents  (D3)  JavaScript  framework  to  depict 
relationships  between  collected  artifacts,  the  portrayal  of  host  anomaly  timelines,  and 
visual  incident  triage  beyond  pure  geospatial  representation. 

Future  toolkit  development  should  pursue  the  inclusion  of  network  policy 
checklists  for  each  site  to  determine  the  logic  used  to  find  anomalies.  To  aid  in 
understanding  potential  adversary  pathways,  the  toolkit  should  also  include  the  collection 
of  or  manual  entry  of  network  interconnect  information,  such  as  firewall  logs,  between 
zones.  It  may  be  possible  to  identify  the  domain  interconnection  type  from  passive 
network  traffic  already  collected,  since  mechanisms  such  as  data  diodes  have  unique 
communication  patterns. 

Additional  offline  databases  of  security  information  could  be  incorporated  beyond 
geolocation  data,  vendor  information,  and  ICS  protocol  identification.  Bro  now  has  an 
offline  autonomous  system  number  (ASN)  lookup  function  to  determine  border  gateway 
protocol  (BGP)  routing  and  Internet  Service  Provider  (ISP)  information. 
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The  malicious  activity  blacklists  and  whitelists  used  were  relatively  small  and 
should  be  expanded.  The  total  number  of  ICS-specific  elements  within  the  current  lists  is 
less  than  200  as  collected  from  observed  ICS  hosts  in  industry  and  online  documentation 
(Figure  20).  ICS  hardware  vendors  could  be  contacted  directly  as  an  authoritative  source 
to  assist  in  compiling  attribute  whitelists  for  their  equipment. 


Figure  20.  Content  distribution  of  initial  ICS-specific  whitelist 

Future  testing  of  the  toolkit  should  be  conducted  on  data  that  includes  known 
malicious  activity  attempts.  To  facilitate  this,  development  of  ICS  network  honeypots 
should  be  pursued  with  tools  like  Conpot,  released  in  2013  by  the  Honeynet  project  to 
simulate  ICS  networks  [74]. 


C.  CONCLUDING  REMARK 

As  the  increasingly  connected  systems  that  power  modem  society’s  critical 
infrastmcture  are  subject  to  more  scmtiny  and  attacks,  those  systems’  owners  and 
operators  will  need  novel  techniques  for  assessing  and  defending  their  ICS  networks.  The 
methodology  presented  in  this  thesis  has  been  tested  and  the  supplementary  toolkit  has 
demonstrated  value.  The  ICS  security  community  would  benefit  from  further  funding  and 
development  of  an  incident  response  toolkit  to  assist  in  securing  our  Nation’s  critical 
assets. 
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